[jira] [Commented] (HDFS-9758) libhdfs++: Implement Python bindings
[
https://issues.apache.org/jira/browse/HDFS-9758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265169#comment-15265169
]
James Clampffer commented on HDFS-9758:
---
{quote}
As you already implemented a reasonable amount of code for CTypes I think it is
best to extend that further.
If you like I can pick up where you left off and finish the remaining calls and
include python tests.
{quote}
I think that would be a really good way to get the bindings started and to get
more people to start using libhdfs++; thanks for taking this on! We can always
get a basic CTypes implementation in and see how it does in practice. If there
are performance issues with ctypes that can be fixed with Cython or you have
the time to do the full blown OO interface initially I'd totally support that
as well.
> libhdfs++: Implement Python bindings
>
>
> Key: HDFS-9758
> URL: https://issues.apache.org/jira/browse/HDFS-9758
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: James Clampffer
> Attachments: hdfs_posix.py
>
>
> It'd be really useful to have bindings for various scripting languages.
> Python would be a good start because of it's popularity and how easy it is to
> interact with shared libraries using the ctypes module. I think bindings for
> the V8 engine that nodeJS uses would be a close second in terms of expanding
> the potential user base.
> Probably worth starting with just adding a synchronous API and building from
> there to avoid interactions with python's garbage collector until the
> bindings prove to be solid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10349) Ozone: StorageContainerManager fails to compile after merge of HDFS-10312 maxDataLength enforcement.
[
https://issues.apache.org/jira/browse/HDFS-10349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-10349:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: HDFS-7240
Status: Resolved (was: Patch Available)
[~arpitagarwal], thank you for the code review. All pre-commit warnings were
unrelated. I have committed this to the HDFS-7240 feature branch.
> Ozone: StorageContainerManager fails to compile after merge of HDFS-10312
> maxDataLength enforcement.
>
>
> Key: HDFS-10349
> URL: https://issues.apache.org/jira/browse/HDFS-10349
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Fix For: HDFS-7240
>
> Attachments: HDFS-10349-HDFS-7240.001.patch
>
>
> HDFS-10312 introduced enforcement of a configurable maximum data length while
> deserializing large block reports. This change broke compilation of
> {{StorageContainerManager}} on the HDFS-7240 feature branch, due to a
> constructor signature change in {{DatanodeProtocolServerSideTranslatorPB}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10351) Ozone: Optimize key writes to chunks by providing a bulk write implementation in ChunkOutputStream.
[
https://issues.apache.org/jira/browse/HDFS-10351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-10351:
-
Attachment: HDFS-10351-HDFS-7240.002.patch
[~anu] and [~jingzhao], thank you for the code reviews.
bq. Technically this can happen even if there is an overflow. But I would
rather have this check and fail the write than otherwise
{{OutputStream}} defines a very specific contract about throwing particular
kinds of runtime exceptions in response to invalid inputs. To ensure adherence
to this contract, I actually adapted the code from OpenJDK's base
{{OutputStream}} class implementation:
http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/file/df209f221cca/src/share/classes/java/io/OutputStream.java#l106
bq. Where are we using this call from ?
This is called from {{org.apache.hadoop.ozone.web.handlers.KeyHandler#putKey}},
which calls {{DistributedStorageHandler}} to get a {{ChunkOutputStream}}, and
then loops reading from the input HTTP request and writing the bytes.
{code}
stream.write(buffer, 0, len);
{code}
Before my patch, this would enter the base class bulk {{OutputStream#write}}
implementation, which is an inefficient loop over the single-byte {{write}}
method. After this patch, this call site instead enters the more efficient
implementation of the bulk {{write}}. This prevents a lot of method call
overhead and allows us to take advantage of the bulk {{ByteBuffer}} operations
for faster transfer.
bq. One minor comment: maybe the following code can be simplified as "int
writeLen = Math.min(CHUNK_SIZE - buffer.position(), len);".
That's a good idea. Here is patch v002 with that change.
> Ozone: Optimize key writes to chunks by providing a bulk write implementation
> in ChunkOutputStream.
> ---
>
> Key: HDFS-10351
> URL: https://issues.apache.org/jira/browse/HDFS-10351
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-10351-HDFS-7240.001.patch,
> HDFS-10351-HDFS-7240.002.patch
>
>
> HDFS-10268 introduced the {{ChunkOutputStream}} class as part of end-to-end
> integration of Ozone receiving key content and writing it to chunks in a
> container. That patch provided an implementation of the mandatory
> single-byte write method. We can improve performance by adding an
> implementation of the bulk write method too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10349) Ozone: StorageContainerManager fails to compile after merge of HDFS-10312 maxDataLength enforcement.
[
https://issues.apache.org/jira/browse/HDFS-10349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265051#comment-15265051
]
Hadoop QA commented on HDFS-10349:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 25s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 3m 52s
{color} | {color:red} root in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 37s
{color} | {color:red} hadoop-hdfs in HDFS-7240 failed with JDK v1.8.0_92.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 36s
{color} | {color:red} hadoop-hdfs in HDFS-7240 failed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 37s
{color} | {color:red} hadoop-hdfs in HDFS-7240 failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 14s
{color} | {color:red} hadoop-hdfs in HDFS-7240 failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 51s
{color} | {color:green} HDFS-7240 passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 59s
{color} | {color:green} HDFS-7240 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 15s {color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_92 with JDK v1.8.0_92
generated 22 new + 13 unchanged - 2 fixed = 35 total (was 15) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 8m 4s {color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95 with JDK v1.7.0_95
generated 21 new + 16 unchanged - 2 fixed = 37 total (was 18) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 46s
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 48s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 29s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_92. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 4s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 197m 54s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_92
[jira] [Commented] (HDFS-10351) Ozone: Optimize key writes to chunks by providing a bulk write implementation in ChunkOutputStream.
[
https://issues.apache.org/jira/browse/HDFS-10351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264993#comment-15264993
]
Hadoop QA commented on HDFS-10351:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 3m 15s
{color} | {color:red} root in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 27s
{color} | {color:red} hadoop-hdfs in HDFS-7240 failed with JDK v1.8.0_91.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 29s
{color} | {color:red} hadoop-hdfs in HDFS-7240 failed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 32s
{color} | {color:red} hadoop-hdfs in HDFS-7240 failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 13s
{color} | {color:red} hadoop-hdfs in HDFS-7240 failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 32s
{color} | {color:green} HDFS-7240 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 33s
{color} | {color:green} HDFS-7240 passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 28s
{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 25s
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_91.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 25s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 30s
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 30s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 19s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new +
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 30s
{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 13s
{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 40s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 32s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 30s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 53s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:cf2ee45 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12801561/HDFS-10351-HDFS-7240.001.patch
|
| JIRA Issue | HDFS-10351 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname |
[jira] [Commented] (HDFS-10351) Ozone: Optimize key writes to chunks by providing a bulk write implementation in ChunkOutputStream.
[
https://issues.apache.org/jira/browse/HDFS-10351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264969#comment-15264969
]
Jing Zhao commented on HDFS-10351:
--
+1 pending Jenkins. One minor comment: maybe the following code can be
simplified as "int writeLen = Math.min(CHUNK_SIZE - buffer.position(), len);".
{code}
117 int targetLen = buffer.position() + len;
118 int writeLen = targetLen <= CHUNK_SIZE ? len :
119 CHUNK_SIZE - buffer.position();
{code}
> Ozone: Optimize key writes to chunks by providing a bulk write implementation
> in ChunkOutputStream.
> ---
>
> Key: HDFS-10351
> URL: https://issues.apache.org/jira/browse/HDFS-10351
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-10351-HDFS-7240.001.patch
>
>
> HDFS-10268 introduced the {{ChunkOutputStream}} class as part of end-to-end
> integration of Ozone receiving key content and writing it to chunks in a
> container. That patch provided an implementation of the mandatory
> single-byte write method. We can improve performance by adding an
> implementation of the bulk write method too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10351) Ozone: Optimize key writes to chunks by providing a bulk write implementation in ChunkOutputStream.
[
https://issues.apache.org/jira/browse/HDFS-10351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264964#comment-15264964
]
Anu Engineer commented on HDFS-10351:
-
+1, pending jenkins.
Two minor comments:
# Technically this can happen even if there is an overflow. But I would rather
have this check and fail the write than otherwise
{{(off + len) < 0)}}
# Where are we using this call from ?
> Ozone: Optimize key writes to chunks by providing a bulk write implementation
> in ChunkOutputStream.
> ---
>
> Key: HDFS-10351
> URL: https://issues.apache.org/jira/browse/HDFS-10351
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-10351-HDFS-7240.001.patch
>
>
> HDFS-10268 introduced the {{ChunkOutputStream}} class as part of end-to-end
> integration of Ozone receiving key content and writing it to chunks in a
> container. That patch provided an implementation of the mandatory
> single-byte write method. We can improve performance by adding an
> implementation of the bulk write method too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10351) Ozone: Optimize key writes to chunks by providing a bulk write implementation in ChunkOutputStream.
[
https://issues.apache.org/jira/browse/HDFS-10351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-10351:
-
Status: Patch Available (was: Open)
> Ozone: Optimize key writes to chunks by providing a bulk write implementation
> in ChunkOutputStream.
> ---
>
> Key: HDFS-10351
> URL: https://issues.apache.org/jira/browse/HDFS-10351
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-10351-HDFS-7240.001.patch
>
>
> HDFS-10268 introduced the {{ChunkOutputStream}} class as part of end-to-end
> integration of Ozone receiving key content and writing it to chunks in a
> container. That patch provided an implementation of the mandatory
> single-byte write method. We can improve performance by adding an
> implementation of the bulk write method too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10351) Ozone: Optimize key writes to chunks by providing a bulk write implementation in ChunkOutputStream.
[
https://issues.apache.org/jira/browse/HDFS-10351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-10351:
-
Attachment: HDFS-10351-HDFS-7240.001.patch
Quoting the JavaDocs of the bulk {{OutputStream#write}} method:
{quote}
The write method of OutputStream calls the write method of one argument on each
of the bytes to be written out. Subclasses are encouraged to override this
method and provide a more efficient implementation.
{quote}
Well, they were sure right! Profiling revealed a ton of function call overhead
from sitting in the base class {{OutputStream#write}}, iterating over the
buffer, and calling the single-byte write method repeatedly. The attached
patch provides a bulk write implementation in the subclass. I have measured a
~30% clock time performance improvement in putting a key. Profiling after the
change shows that this isn't a hotspot anymore.
There are no new tests in this patch. I'm relying on the existing tests or
regression testing, and I'm relying on the performance measure/profiling I
described above as proof that the optimization worked.
> Ozone: Optimize key writes to chunks by providing a bulk write implementation
> in ChunkOutputStream.
> ---
>
> Key: HDFS-10351
> URL: https://issues.apache.org/jira/browse/HDFS-10351
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-10351-HDFS-7240.001.patch
>
>
> HDFS-10268 introduced the {{ChunkOutputStream}} class as part of end-to-end
> integration of Ozone receiving key content and writing it to chunks in a
> container. That patch provided an implementation of the mandatory
> single-byte write method. We can improve performance by adding an
> implementation of the bulk write method too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10351) Ozone: Optimize key writes to chunks by providing a bulk write implementation in ChunkOutputStream.
Chris Nauroth created HDFS-10351:
Summary: Ozone: Optimize key writes to chunks by providing a bulk
write implementation in ChunkOutputStream.
Key: HDFS-10351
URL: https://issues.apache.org/jira/browse/HDFS-10351
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Reporter: Chris Nauroth
Assignee: Chris Nauroth
HDFS-10268 introduced the {{ChunkOutputStream}} class as part of end-to-end
integration of Ozone receiving key content and writing it to chunks in a
container. That patch provided an implementation of the mandatory single-byte
write method. We can improve performance by adding an implementation of the
bulk write method too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10349) Ozone: StorageContainerManager fails to compile after merge of HDFS-10312 maxDataLength enforcement.
[
https://issues.apache.org/jira/browse/HDFS-10349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-10349:
-
Summary: Ozone: StorageContainerManager fails to compile after merge of
HDFS-10312 maxDataLength enforcement. (was: StorageContainerManager fails to
compile after merge of HDFS-10312 maxDataLength enforcement.)
> Ozone: StorageContainerManager fails to compile after merge of HDFS-10312
> maxDataLength enforcement.
>
>
> Key: HDFS-10349
> URL: https://issues.apache.org/jira/browse/HDFS-10349
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-10349-HDFS-7240.001.patch
>
>
> HDFS-10312 introduced enforcement of a configurable maximum data length while
> deserializing large block reports. This change broke compilation of
> {{StorageContainerManager}} on the HDFS-7240 feature branch, due to a
> constructor signature change in {{DatanodeProtocolServerSideTranslatorPB}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10350) Implement asynchronous setOwner for DistributedFileSystem
Xiaobing Zhou created HDFS-10350: Summary: Implement asynchronous setOwner for DistributedFileSystem Key: HDFS-10350 URL: https://issues.apache.org/jira/browse/HDFS-10350 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Xiaobing Zhou Assignee: Xiaobing Zhou -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10350) Implement asynchronous setOwner for DistributedFileSystem
[ https://issues.apache.org/jira/browse/HDFS-10350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-10350: - Description: This is proposed to implement an asynchronous setOwner. > Implement asynchronous setOwner for DistributedFileSystem > - > > Key: HDFS-10350 > URL: https://issues.apache.org/jira/browse/HDFS-10350 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs, hdfs-client >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > > This is proposed to implement an asynchronous setOwner. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10328) Add cache pool level replication managment
[ https://issues.apache.org/jira/browse/HDFS-10328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264841#comment-15264841 ] Colin Patrick McCabe commented on HDFS-10328: - Hi [~xupener], Interesting idea. However, this doesn't sound like "cache pool level replication management", since the replication management is still per-directive, even after this patch. This seems like adding a per-cache-pool default. If you agree, can you update the JIRA name and some of the names in the patch? > Add cache pool level replication managment > --- > > Key: HDFS-10328 > URL: https://issues.apache.org/jira/browse/HDFS-10328 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching >Reporter: xupeng >Assignee: xupeng >Priority: Minor > Fix For: 2.3.0 > > Attachments: HDFS-10328.001.patch > > > For now, hdfs cacheadmin can not set a replication num for cachepool. Each > cache directive added in the cache pool should set their own replication num > individually. > Consider this situation, we add daily hive table into cache pool "hive" .Each > time i should set the same replication num for every table directive in the > same cache pool. > I think we should enable setting a replication num for cachepool that every > cache directive in the pool can inherit replication configuration from the > pool. Also cache directive can override replication configuration explicitly > by calling "add & modify directive -replication" command from cli. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10349) StorageContainerManager fails to compile after merge of HDFS-10312 maxDataLength enforcement.
[
https://issues.apache.org/jira/browse/HDFS-10349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264825#comment-15264825
]
Arpit Agarwal commented on HDFS-10349:
--
+1 thank you Chris. My merge from trunk must have broken this. I am sorry about
that.
> StorageContainerManager fails to compile after merge of HDFS-10312
> maxDataLength enforcement.
> -
>
> Key: HDFS-10349
> URL: https://issues.apache.org/jira/browse/HDFS-10349
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-10349-HDFS-7240.001.patch
>
>
> HDFS-10312 introduced enforcement of a configurable maximum data length while
> deserializing large block reports. This change broke compilation of
> {{StorageContainerManager}} on the HDFS-7240 feature branch, due to a
> constructor signature change in {{DatanodeProtocolServerSideTranslatorPB}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10349) StorageContainerManager fails to compile after merge of HDFS-10312 maxDataLength enforcement.
[
https://issues.apache.org/jira/browse/HDFS-10349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-10349:
-
Attachment: HDFS-10349-HDFS-7240.001.patch
[~arpitagarwal], would you please code review? Thank you.
> StorageContainerManager fails to compile after merge of HDFS-10312
> maxDataLength enforcement.
> -
>
> Key: HDFS-10349
> URL: https://issues.apache.org/jira/browse/HDFS-10349
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-10349-HDFS-7240.001.patch
>
>
> HDFS-10312 introduced enforcement of a configurable maximum data length while
> deserializing large block reports. This change broke compilation of
> {{StorageContainerManager}} on the HDFS-7240 feature branch, due to a
> constructor signature change in {{DatanodeProtocolServerSideTranslatorPB}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10349) StorageContainerManager fails to compile after merge of HDFS-10312 maxDataLength enforcement.
[
https://issues.apache.org/jira/browse/HDFS-10349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-10349:
-
Status: Patch Available (was: Open)
> StorageContainerManager fails to compile after merge of HDFS-10312
> maxDataLength enforcement.
> -
>
> Key: HDFS-10349
> URL: https://issues.apache.org/jira/browse/HDFS-10349
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-10349-HDFS-7240.001.patch
>
>
> HDFS-10312 introduced enforcement of a configurable maximum data length while
> deserializing large block reports. This change broke compilation of
> {{StorageContainerManager}} on the HDFS-7240 feature branch, due to a
> constructor signature change in {{DatanodeProtocolServerSideTranslatorPB}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10349) StorageContainerManager fails to compile after merge of HDFS-10312 maxDataLength enforcement.
Chris Nauroth created HDFS-10349:
Summary: StorageContainerManager fails to compile after merge of
HDFS-10312 maxDataLength enforcement.
Key: HDFS-10349
URL: https://issues.apache.org/jira/browse/HDFS-10349
Project: Hadoop HDFS
Issue Type: Bug
Components: ozone
Reporter: Chris Nauroth
Assignee: Chris Nauroth
HDFS-10312 introduced enforcement of a configurable maximum data length while
deserializing large block reports. This change broke compilation of
{{StorageContainerManager}} on the HDFS-7240 feature branch, due to a
constructor signature change in {{DatanodeProtocolServerSideTranslatorPB}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10349) StorageContainerManager fails to compile after merge of HDFS-10312 maxDataLength enforcement.
[
https://issues.apache.org/jira/browse/HDFS-10349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-10349:
-
Issue Type: Sub-task (was: Bug)
Parent: HDFS-7240
> StorageContainerManager fails to compile after merge of HDFS-10312
> maxDataLength enforcement.
> -
>
> Key: HDFS-10349
> URL: https://issues.apache.org/jira/browse/HDFS-10349
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>
> HDFS-10312 introduced enforcement of a configurable maximum data length while
> deserializing large block reports. This change broke compilation of
> {{StorageContainerManager}} on the HDFS-7240 feature branch, due to a
> constructor signature change in {{DatanodeProtocolServerSideTranslatorPB}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10347) Namenode report bad block method doesn't log the bad block or datanode.
[
https://issues.apache.org/jira/browse/HDFS-10347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264785#comment-15264785
]
Mingliang Liu commented on HDFS-10347:
--
If we are locking/unlocking for each reported block/node instead of the whole
RPC request, it will be much easier. I agree that the overhead is not big deal
here. Thanks for the patch.
> Namenode report bad block method doesn't log the bad block or datanode.
> ---
>
> Key: HDFS-10347
> URL: https://issues.apache.org/jira/browse/HDFS-10347
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Minor
> Fix For: 2.7.3
>
> Attachments: HDFS-10347.patch
>
>
> Currently the method {{FSNamesystem#reportBadBlocks}} doesn't log any
> information regarding the bad block id or the datanode on which corrupt block
> is detected.
> It would be helpful to log that information to debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10301) BlockReport retransmissions may lead to storages falsely being declared zombie if storage report processing happens out of order
[ https://issues.apache.org/jira/browse/HDFS-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264772#comment-15264772 ] Colin Patrick McCabe commented on HDFS-10301: - bq. You can think of it as a new operation SyncStorages, which does just that - updates NameNode's knowledge of DN's storages. I combined this operation with the first br-RPC. One can combine it with any other call, same as you propose to combine it with the heartbeat. Except it seems a poor idea, since we don't want to wait for removal of thousands of replicas on a heartbeat. Thanks for explaining your proposal a little bit more. I agree that enumerating all the storages in the first block report RPC is a fairly simple way to handle this, and shouldn't add too much size to the FBR. It seems like a better idea than adding it to the heartbeat, like I proposed. In the short term, however, I would prefer the current patch, since it involves no RPC changes, and doesn't require all the DataNodes to be upgraded before it can work. > BlockReport retransmissions may lead to storages falsely being declared > zombie if storage report processing happens out of order > > > Key: HDFS-10301 > URL: https://issues.apache.org/jira/browse/HDFS-10301 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.1 >Reporter: Konstantin Shvachko >Assignee: Colin Patrick McCabe >Priority: Critical > Attachments: HDFS-10301.002.patch, HDFS-10301.003.patch, > HDFS-10301.01.patch, zombieStorageLogs.rtf > > > When NameNode is busy a DataNode can timeout sending a block report. Then it > sends the block report again. Then NameNode while process these two reports > at the same time can interleave processing storages from different reports. > This screws up the blockReportId field, which makes NameNode think that some > storages are zombie. Replicas from zombie storages are immediately removed, > causing missing blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10347) Namenode report bad block method doesn't log the bad block or datanode.
[
https://issues.apache.org/jira/browse/HDFS-10347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264768#comment-15264768
]
Rushabh S Shah commented on HDFS-10347:
---
Thanks [~kihwal] for reviewing and committing.
Thanks [~liuml07] for additional reviews. :)
> Namenode report bad block method doesn't log the bad block or datanode.
> ---
>
> Key: HDFS-10347
> URL: https://issues.apache.org/jira/browse/HDFS-10347
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Minor
> Fix For: 2.7.3
>
> Attachments: HDFS-10347.patch
>
>
> Currently the method {{FSNamesystem#reportBadBlocks}} doesn't log any
> information regarding the bad block id or the datanode on which corrupt block
> is detected.
> It would be helpful to log that information to debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10347) Namenode report bad block method doesn't log the bad block or datanode.
[
https://issues.apache.org/jira/browse/HDFS-10347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264766#comment-15264766
]
Rushabh S Shah commented on HDFS-10347:
---
There are 2 approaches:
1. Go through the array of LocatedBlock outside the lock and print every block
and then again go through that array to mark them corrupt.
2. Go through array only once and take a negligible hit for logging within
write lock.
We are not looking up any map or data structure so I think the impact is almost
negligible.
Any thoughts ?
> Namenode report bad block method doesn't log the bad block or datanode.
> ---
>
> Key: HDFS-10347
> URL: https://issues.apache.org/jira/browse/HDFS-10347
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Minor
> Fix For: 2.7.3
>
> Attachments: HDFS-10347.patch
>
>
> Currently the method {{FSNamesystem#reportBadBlocks}} doesn't log any
> information regarding the bad block id or the datanode on which corrupt block
> is detected.
> It would be helpful to log that information to debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10347) Namenode report bad block method doesn't log the bad block or datanode.
[
https://issues.apache.org/jira/browse/HDFS-10347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264764#comment-15264764
]
Hudson commented on HDFS-10347:
---
SUCCESS: Integrated in Hadoop-trunk-Commit #9697 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9697/])
HDFS-10347. Namenode report bad block method doesn't log the bad block (kihwal:
rev 7da540d03eccc2f97950bf47e8b35ce8c889d1e0)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
> Namenode report bad block method doesn't log the bad block or datanode.
> ---
>
> Key: HDFS-10347
> URL: https://issues.apache.org/jira/browse/HDFS-10347
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Minor
> Fix For: 2.7.3
>
> Attachments: HDFS-10347.patch
>
>
> Currently the method {{FSNamesystem#reportBadBlocks}} doesn't log any
> information regarding the bad block id or the datanode on which corrupt block
> is detected.
> It would be helpful to log that information to debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10347) Namenode report bad block method doesn't log the bad block or datanode.
[
https://issues.apache.org/jira/browse/HDFS-10347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-10347:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.7.3
Status: Resolved (was: Patch Available)
> Namenode report bad block method doesn't log the bad block or datanode.
> ---
>
> Key: HDFS-10347
> URL: https://issues.apache.org/jira/browse/HDFS-10347
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Minor
> Fix For: 2.7.3
>
> Attachments: HDFS-10347.patch
>
>
> Currently the method {{FSNamesystem#reportBadBlocks}} doesn't log any
> information regarding the bad block id or the datanode on which corrupt block
> is detected.
> It would be helpful to log that information to debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10347) Namenode report bad block method doesn't log the bad block or datanode.
[
https://issues.apache.org/jira/browse/HDFS-10347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264749#comment-15264749
]
Kihwal Lee commented on HDFS-10347:
---
Committed this to trunk through branch-2.7. Thanks for improving this, Rushabh.
> Namenode report bad block method doesn't log the bad block or datanode.
> ---
>
> Key: HDFS-10347
> URL: https://issues.apache.org/jira/browse/HDFS-10347
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Minor
> Attachments: HDFS-10347.patch
>
>
> Currently the method {{FSNamesystem#reportBadBlocks}} doesn't log any
> information regarding the bad block id or the datanode on which corrupt block
> is detected.
> It would be helpful to log that information to debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10347) Namenode report bad block method doesn't log the bad block or datanode.
[
https://issues.apache.org/jira/browse/HDFS-10347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264747#comment-15264747
]
Mingliang Liu commented on HDFS-10347:
--
+1 (non-binding) as the patch is simple and clear.
My minor concern is we're dumping state change log while holding the lock,
which is not ideal.
> Namenode report bad block method doesn't log the bad block or datanode.
> ---
>
> Key: HDFS-10347
> URL: https://issues.apache.org/jira/browse/HDFS-10347
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Minor
> Attachments: HDFS-10347.patch
>
>
> Currently the method {{FSNamesystem#reportBadBlocks}} doesn't log any
> information regarding the bad block id or the datanode on which corrupt block
> is detected.
> It would be helpful to log that information to debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10347) Namenode report bad block method doesn't log the bad block or datanode.
[
https://issues.apache.org/jira/browse/HDFS-10347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264712#comment-15264712
]
Kihwal Lee edited comment on HDFS-10347 at 4/29/16 8:56 PM:
+1 straightforward logging change.
I ran {{TestProcessCorruptBlocks}} and saw this in the log.
{noformat}
*DIR* reportBadBlocks for block:
BP-1292934650-x.x.x.x-1461963215660:blk_1073741825_1001
on datanode: 127.0.0.1:41290
{noformat}
was (Author: kihwal):
+1 straightforward logging change.
> Namenode report bad block method doesn't log the bad block or datanode.
> ---
>
> Key: HDFS-10347
> URL: https://issues.apache.org/jira/browse/HDFS-10347
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Minor
> Attachments: HDFS-10347.patch
>
>
> Currently the method {{FSNamesystem#reportBadBlocks}} doesn't log any
> information regarding the bad block id or the datanode on which corrupt block
> is detected.
> It would be helpful to log that information to debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10347) Namenode report bad block method doesn't log the bad block or datanode.
[
https://issues.apache.org/jira/browse/HDFS-10347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264712#comment-15264712
]
Kihwal Lee commented on HDFS-10347:
---
+1 straightforward logging change.
> Namenode report bad block method doesn't log the bad block or datanode.
> ---
>
> Key: HDFS-10347
> URL: https://issues.apache.org/jira/browse/HDFS-10347
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Minor
> Attachments: HDFS-10347.patch
>
>
> Currently the method {{FSNamesystem#reportBadBlocks}} doesn't log any
> information regarding the bad block id or the datanode on which corrupt block
> is detected.
> It would be helpful to log that information to debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-334) dfsadmin -metasave should also log corrupt replicas info
[ https://issues.apache.org/jira/browse/HDFS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264692#comment-15264692 ] Denis Bolshakov commented on HDFS-334: -- [~boky01]Looks like I am too late. I've just the following https://issues.apache.org/jira/browse/HDFS-10330 Seems the same issue and it's in master branch since Wed Apr 27 08:19:48 2016 -0500. So this ticker could be closed as duplicate. Can you handle this? > dfsadmin -metasave should also log corrupt replicas info > > > Key: HDFS-334 > URL: https://issues.apache.org/jira/browse/HDFS-334 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Lohit Vijayarenu >Assignee: Denis Bolshakov >Priority: Minor > Labels: newbie > Attachments: HDFS-334.001.patch, HDFS-334.002.patch, > HDFS-334.003.patch > > > _hadoop dfsadmin -metasave _ should also dump information about > corrupt replicas map. This could help in telling if pending replication was > due to corrupt replicas. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10347) Namenode report bad block method doesn't log the bad block or datanode.
[
https://issues.apache.org/jira/browse/HDFS-10347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264671#comment-15264671
]
Hadoop QA commented on HDFS-10347:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
59s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} trunk passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} trunk passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 49s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 40s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 58m 29s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 55m 26s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 140m 12s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_91 Failed junit tests | hadoop.hdfs.TestHFlush |
| JDK v1.7.0_95 Failed junit tests | hadoop.hdfs.TestHFlush |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:cf2ee45 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12801489/HDFS-10347.patch |
| JIRA Issue | HDFS-10347 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux a4578694edda
[jira] [Commented] (HDFS-3702) Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client
[ https://issues.apache.org/jira/browse/HDFS-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264590#comment-15264590 ] stack commented on HDFS-3702: - Any chance of getting this on 2.8 branch? Thanks. > Add an option for NOT writing the blocks locally if there is a datanode on > the same box as the client > - > > Key: HDFS-3702 > URL: https://issues.apache.org/jira/browse/HDFS-3702 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 2.5.1 >Reporter: Nicolas Liochon >Assignee: Lei (Eddy) Xu >Priority: Minor > Labels: BB2015-05-TBR > Fix For: 3.0.0, 2.9.0 > > Attachments: HDFS-3702.000.patch, HDFS-3702.001.patch, > HDFS-3702.002.patch, HDFS-3702.003.patch, HDFS-3702.004.patch, > HDFS-3702.005.patch, HDFS-3702.006.patch, HDFS-3702.007.patch, > HDFS-3702.008.patch, HDFS-3702.009.patch, HDFS-3702.010.patch, > HDFS-3702.011.patch, HDFS-3702.012.patch, HDFS-3702_Design.pdf > > > This is useful for Write-Ahead-Logs: these files are writen for recovery > only, and are not read when there are no failures. > Taking HBase as an example, these files will be read only if the process that > wrote them (the 'HBase regionserver') dies. This will likely come from a > hardware failure, hence the corresponding datanode will be dead as well. So > we're writing 3 replicas, but in reality only 2 of them are really useful. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10348) Namenode report bad block method doesn't check whether the block belongs to datanode before adding it to corrupt replicas map.
[
https://issues.apache.org/jira/browse/HDFS-10348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HDFS-10348:
--
Description:
Namenode (via report bad block nethod) doesn't check whether the block belongs
to the datanode before it adds to corrupt replicas map.
In one of our cluster we found that there were 3 lingering corrupt blocks.
It happened in the following order.
1. Two clients called getBlockLocations for a particular file.
2. Client C1 tried to open the file and encountered checksum error from node
N3 and it reported bad block (blk1) to the namenode.
3. Namenode added that node N3 and block blk1 to corrrupt replicas map and
ask one of the good node (one of the 2 nodes) to replicate the block to another
node N4.
4. After receiving the block, N4 sends an IBR (with RECEIVED_BLOCK) to namenode.
5. Namenode removed the block and node N3 from corrupt replicas map.
It also removed N3's storage from triplets and queued an invalidate request
for N3.
6. In the mean time, Client C2 tries to open the file and the request went to
node N3.
C2 also encountered the checksum exception and reported bad block to
namenode.
7. Namenode added the corrupt block blk1 and node N3 to the corrupt replicas
map without confirming whether node N3 has the block or not.
After deleting the block, N3 sends an IBR (with DELETED) and the namenode
simply ignores the report since the N3's storage is no longer in the
triplets(from step 5)
We took the node out of rotation, but still the block was present only in the
corruptReplciasMap.
Since on removing the node, we only goes through the block which are present in
the triplets for a given datanode.
[~kshukla]'s patch fixed this bug via
https://issues.apache.org/jira/browse/HDFS-9958.
But I think the following check should be made in the
BlockManager#markBlockAsCorrupt instead of
BlockManager#findAndMarkBlockAsCorrupt.
{noformat}
if (storage == null) {
storage = storedBlock.findStorageInfo(node);
}
if (storage == null) {
blockLog.debug("BLOCK* findAndMarkBlockAsCorrupt: {} not found on {}",
blk, dn);
return;
}
{noformat}
was:
Namenode (via report bad block nethod) doesn't check whether the block belongs
to the datanode before it adds to corrupt replicas map.
In one of our cluster we found that there were 3 lingering corrupt blocks.
It happened in the following order.
1. Two clients called getBlockLocations for a particular file.
2. Client C1 tried to open the file and encountered checksum error from node
N3 and it reported bad block (blk1) to the namenode.
3. Namenode added that node N3 and block blk1 to corrrupt replicas map and
ask one of the good node (one of the 2 nodes) to replicate the block to another
node N4.
4. After receiving the block, N4 sends an IBR (with RECEIVED_BLOCK) to namenode.
5. Namenode removed the block and node N3 from corrupt replicas map.
It also removed N3's storage from triplets and queued an invalidate request
for N3.
6. In the mean time, Client C2 tries to open the file and the request went to
node N3.
C2 also encountered the checksum exception and reported bad block to
namenode.
7. Namenode added the corrupt block blk1 and node N3 to the corrupt replicas
map without confirming whether node N3 has the block or not.
After deleting the block, N3 sends an IBR (with DELETED) and the namenode
simply ignores the report since the N3's storage is no longer in the
triplets(from step 5)
We took the node out of rotation, but still the block was present only in the
corruptReplciasMap.
Since on removing the node, we only goes through the block which are present in
the triplets for a given datanode.
Kuhu's patch fixed this bug via https://issues.apache.org/jira/browse/HDFS-9958.
But I think the following check should be made in the
BlockManager#markBlockAsCorrupt instead of
BlockManager#findAndMarkBlockAsCorrupt.
{noformat}
if (storage == null) {
storage = storedBlock.findStorageInfo(node);
}
if (storage == null) {
blockLog.debug("BLOCK* findAndMarkBlockAsCorrupt: {} not found on {}",
blk, dn);
return;
}
{noformat}
> Namenode report bad block method doesn't check whether the block belongs to
> datanode before adding it to corrupt replicas map.
> --
>
> Key: HDFS-10348
> URL: https://issues.apache.org/jira/browse/HDFS-10348
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>
> Namenode (via report bad block nethod) doesn't check whether the block
> belongs to the datanode before it adds to corrupt replicas map.
> In one of our cluster we found that
[jira] [Created] (HDFS-10348) Namenode report bad block method doesn't check whether the block belongs to datanode before adding it to corrupt replicas map.
Rushabh S Shah created HDFS-10348:
-
Summary: Namenode report bad block method doesn't check whether
the block belongs to datanode before adding it to corrupt replicas map.
Key: HDFS-10348
URL: https://issues.apache.org/jira/browse/HDFS-10348
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.7.0
Reporter: Rushabh S Shah
Assignee: Rushabh S Shah
Namenode (via report bad block nethod) doesn't check whether the block belongs
to the datanode before it adds to corrupt replicas map.
In one of our cluster we found that there were 3 lingering corrupt blocks.
It happened in the following order.
1. Two clients called getBlockLocations for a particular file.
2. Client C1 tried to open the file and encountered checksum error from node
N3 and it reported bad block (blk1) to the namenode.
3. Namenode added that node N3 and block blk1 to corrrupt replicas map and
ask one of the good node (one of the 2 nodes) to replicate the block to another
node N4.
4. After receiving the block, N4 sends an IBR (with RECEIVED_BLOCK) to namenode.
5. Namenode removed the block and node N3 from corrupt replicas map.
It also removed N3's storage from triplets and queued an invalidate request
for N3.
6. In the mean time, Client C2 tries to open the file and the request went to
node N3.
C2 also encountered the checksum exception and reported bad block to
namenode.
7. Namenode added the corrupt block blk1 and node N3 to the corrupt replicas
map without confirming whether node N3 has the block or not.
After deleting the block, N3 sends an IBR (with DELETED) and the namenode
simply ignores the report since the N3's storage is no longer in the
triplets(from step 5)
We took the node out of rotation, but still the block was present only in the
corruptReplciasMap.
Since on removing the node, we only goes through the block which are present in
the triplets for a given datanode.
Kuhu's patch fixed this bug via https://issues.apache.org/jira/browse/HDFS-9958.
But I think the following check should be made in the
BlockManager#markBlockAsCorrupt instead of
BlockManager#findAndMarkBlockAsCorrupt.
{noformat}
if (storage == null) {
storage = storedBlock.findStorageInfo(node);
}
if (storage == null) {
blockLog.debug("BLOCK* findAndMarkBlockAsCorrupt: {} not found on {}",
blk, dn);
return;
}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-334) dfsadmin -metasave should also log corrupt replicas info
[ https://issues.apache.org/jira/browse/HDFS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264558#comment-15264558 ] Andras Bokor commented on HDFS-334: --- [~bolshakov.de...@gmail.com] Why? You wrote the test failures are not related. I think at this point (if you are sure the test failures are not related) you need to find a reviewer who can accept/reject/commit your patch. Note that there are a lot of intermittent test failures in Hadoop, so this does not always mean wrong patch. > dfsadmin -metasave should also log corrupt replicas info > > > Key: HDFS-334 > URL: https://issues.apache.org/jira/browse/HDFS-334 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Lohit Vijayarenu >Assignee: Denis Bolshakov >Priority: Minor > Labels: newbie > Attachments: HDFS-334.001.patch, HDFS-334.002.patch, > HDFS-334.003.patch > > > _hadoop dfsadmin -metasave _ should also dump information about > corrupt replicas map. This could help in telling if pending replication was > due to corrupt replicas. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9389) Add maintenance states to AdminStates
[
https://issues.apache.org/jira/browse/HDFS-9389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264534#comment-15264534
]
Hadoop QA commented on HDFS-9389:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 18s
{color} | {color:green} trunk passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 20s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 26s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s
{color} | {color:green} trunk passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 10s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 24s
{color} | {color:red} hadoop-hdfs-project: patch generated 3 new + 90 unchanged
- 0 fixed = 93 total (was 90) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 6s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 50s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 58m 22s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 56s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 55m 21s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} |
[jira] [Commented] (HDFS-10335) Mover$Processor#chooseTarget() always chooses the first matching target storage group
[
https://issues.apache.org/jira/browse/HDFS-10335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264503#comment-15264503
]
Hudson commented on HDFS-10335:
---
FAILURE: Integrated in Hadoop-trunk-Commit #9695 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9695/])
HDFS-10335 Mover$Processor#chooseTarget() always chooses the first (szetszwo:
rev 4da6f69ca129b28a5dad0a66d0c24e725ce25a3a)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java
> Mover$Processor#chooseTarget() always chooses the first matching target
> storage group
> -
>
> Key: HDFS-10335
> URL: https://issues.apache.org/jira/browse/HDFS-10335
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Critical
> Fix For: 2.7.3
>
> Attachments: HDFS-10335.000.patch, HDFS-10335.000.patch
>
>
> Currently the
> {{org.apache.hadoop.hdfs.server.mover.Mover$Processor#chooseTarget()}} always
> chooses the first matching target datanode from the candidate list. This may
> make the mover schedule a lot of task to a few of the datanodes (first
> several datanodes of the candidate list). The overall performance will suffer
> significantly from this because of the saturated network/disk usage.
> Specially, if the {{dfs.datanode.balance.max.concurrent.moves}} is set, the
> scheduled move task will be queued on a few of the storage group, regardless
> of other available storage groups. We need an algorithm which can distribute
> the move tasks approximately even across all the candidate target storage
> groups.
> Thanks [~szetszwo] for offline discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10335) Mover$Processor#chooseTarget() always chooses the first matching target storage group
[
https://issues.apache.org/jira/browse/HDFS-10335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264494#comment-15264494
]
Mingliang Liu commented on HDFS-10335:
--
Thanks for your review and commit, [~szetszwo].
> Mover$Processor#chooseTarget() always chooses the first matching target
> storage group
> -
>
> Key: HDFS-10335
> URL: https://issues.apache.org/jira/browse/HDFS-10335
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Critical
> Fix For: 2.7.3
>
> Attachments: HDFS-10335.000.patch, HDFS-10335.000.patch
>
>
> Currently the
> {{org.apache.hadoop.hdfs.server.mover.Mover$Processor#chooseTarget()}} always
> chooses the first matching target datanode from the candidate list. This may
> make the mover schedule a lot of task to a few of the datanodes (first
> several datanodes of the candidate list). The overall performance will suffer
> significantly from this because of the saturated network/disk usage.
> Specially, if the {{dfs.datanode.balance.max.concurrent.moves}} is set, the
> scheduled move task will be queued on a few of the storage group, regardless
> of other available storage groups. We need an algorithm which can distribute
> the move tasks approximately even across all the candidate target storage
> groups.
> Thanks [~szetszwo] for offline discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10335) Mover$Processor#chooseTarget() always chooses the first matching target storage group
[
https://issues.apache.org/jira/browse/HDFS-10335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo Nicholas Sze updated HDFS-10335:
---
Resolution: Fixed
Fix Version/s: 2.7.3
Status: Resolved (was: Patch Available)
I have committed this. Thanks, Mingliang!
> Mover$Processor#chooseTarget() always chooses the first matching target
> storage group
> -
>
> Key: HDFS-10335
> URL: https://issues.apache.org/jira/browse/HDFS-10335
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Critical
> Fix For: 2.7.3
>
> Attachments: HDFS-10335.000.patch, HDFS-10335.000.patch
>
>
> Currently the
> {{org.apache.hadoop.hdfs.server.mover.Mover$Processor#chooseTarget()}} always
> chooses the first matching target datanode from the candidate list. This may
> make the mover schedule a lot of task to a few of the datanodes (first
> several datanodes of the candidate list). The overall performance will suffer
> significantly from this because of the saturated network/disk usage.
> Specially, if the {{dfs.datanode.balance.max.concurrent.moves}} is set, the
> scheduled move task will be queued on a few of the storage group, regardless
> of other available storage groups. We need an algorithm which can distribute
> the move tasks approximately even across all the candidate target storage
> groups.
> Thanks [~szetszwo] for offline discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10347) Namenode report bad block method doesn't log the bad block or datanode.
[
https://issues.apache.org/jira/browse/HDFS-10347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HDFS-10347:
--
Status: Patch Available (was: Open)
> Namenode report bad block method doesn't log the bad block or datanode.
> ---
>
> Key: HDFS-10347
> URL: https://issues.apache.org/jira/browse/HDFS-10347
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Minor
> Attachments: HDFS-10347.patch
>
>
> Currently the method {{FSNamesystem#reportBadBlocks}} doesn't log any
> information regarding the bad block id or the datanode on which corrupt block
> is detected.
> It would be helpful to log that information to debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10347) Namenode report bad block method doesn't log the bad block or datanode.
[
https://issues.apache.org/jira/browse/HDFS-10347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HDFS-10347:
--
Attachment: HDFS-10347.patch
> Namenode report bad block method doesn't log the bad block or datanode.
> ---
>
> Key: HDFS-10347
> URL: https://issues.apache.org/jira/browse/HDFS-10347
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Minor
> Attachments: HDFS-10347.patch
>
>
> Currently the method {{FSNamesystem#reportBadBlocks}} doesn't log any
> information regarding the bad block id or the datanode on which corrupt block
> is detected.
> It would be helpful to log that information to debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10340) data node sudden killed
[ https://issues.apache.org/jira/browse/HDFS-10340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264195#comment-15264195 ] Kihwal Lee commented on HDFS-10340: --- Then it could be linux OOM killer. > data node sudden killed > > > Key: HDFS-10340 > URL: https://issues.apache.org/jira/browse/HDFS-10340 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 > Environment: Ubuntu 16.04 LTS , RAM 16g , cpu core : 8 , hdd 100gb, > hadoop 2.6.0 >Reporter: tu nguyen khac >Priority: Critical > > I tried to setup a new data node using ubuntu 16 > and get it join to an existed Hadoop Hdfs cluster ( there are 10 nodes in > this cluster and they all run on centos Os 6 ) > But when i try to boostrap this node , after about 10 or 20 minutes i get > this strange errors : > 2016-04-26 20:12:09,394 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: > /10.3.24.65:55323, dest: /10.3.24.197:50010, bytes: 79902, op: HDFS_WRITE, > cliID: DFSClient_NONMAPREDUCE_1379996362_1, offset: 0, srvID: > 225f5b43-1dd3-4ac6-88d2-1e8d27dba55b, blockid: > BP-352432948-10.3.24.65-1433821675295:blk_1074038505_789832, duration: > 15331628 > 2016-04-26 20:12:09,394 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > PacketResponder: BP-352432948-10.3.24.65-1433821675295:blk_1074038505_789832, > type=LAST_IN_PIPELINE, downstreams=0:[] terminating > 2016-04-26 20:12:25,410 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for BP-352432948-10.3.24.65-1433821675295:blk_1074038502_789829 > 2016-04-26 20:12:25,411 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for BP-352432948-10.3.24.65-1433821675295:blk_1074038505_789832 > 2016-04-26 20:13:18,546 INFO > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: > Scheduling blk_1074038502_789829 file > /data/hadoop_data/backup/data/current/BP-352432948-10.3.24.65-1433821675295/current/finalized/subdir4/subdir134/blk_1074038502 > for deletion > 2016-04-26 20:13:18,562 INFO > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: > Deleted BP-352432948-10.3.24.65-1433821675295 blk_1074038502_789829 file > /data/hadoop_data/backup/data/current/BP-352432948-10.3.24.65-1433821675295/current/finalized/subdir4/subdir134/blk_1074038502 > 2016-04-26 20:15:46,481 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: RECEIVED SIGNAL 15: SIGTERM > 2016-04-26 20:15:46,504 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > SHUTDOWN_MSG: > / > SHUTDOWN_MSG: Shutting down DataNode at bigdata-dw-24-197/10.3.24.197 > / -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10347) Namenode report bad block method doesn't log the bad block or datanode.
[
https://issues.apache.org/jira/browse/HDFS-10347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-10347:
--
Summary: Namenode report bad block method doesn't log the bad block or
datanode. (was: Namenode report bad bad method doesn't log the bad block or
datanode.)
> Namenode report bad block method doesn't log the bad block or datanode.
> ---
>
> Key: HDFS-10347
> URL: https://issues.apache.org/jira/browse/HDFS-10347
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Minor
>
> Currently the method {{FSNamesystem#reportBadBlocks}} doesn't log any
> information regarding the bad block id or the datanode on which corrupt block
> is detected.
> It would be helpful to log that information to debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10260) TestFsDatasetImpl#testCleanShutdownOfVolume often fails
[
https://issues.apache.org/jira/browse/HDFS-10260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264178#comment-15264178
]
Hudson commented on HDFS-10260:
---
FAILURE: Integrated in Hadoop-trunk-Commit #9693 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9693/])
HDFS-10260. TestFsDatasetImpl#testCleanShutdownOfVolume often fails. (kihwal:
rev af9b000535cc987f66327e2b2bfe08923ba24c13)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
> TestFsDatasetImpl#testCleanShutdownOfVolume often fails
> ---
>
> Key: HDFS-10260
> URL: https://issues.apache.org/jira/browse/HDFS-10260
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, test
>Reporter: Wei-Chiu Chuang
>Assignee: Rushabh S Shah
> Fix For: 2.8.0
>
> Attachments: HDFS-10260-v1.patch, HDFS-10260.patch
>
>
> This test failure occurs in upstream Jenkins. Looking at the test code, I
> think it should be improved to capture the root cause of failure:
> E.g. change {{Thread.sleep(1000)}} to {{GenericTestUtils.waitFor}} and use
> {{GenericTestUtils.assertExceptionContains}} to replace
> {code}
> Assert.assertTrue(ioe.getMessage().contains(info.toString()));
> {code}
> https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/1062/testReport/junit/org.apache.hadoop.hdfs.server.datanode.fsdataset.impl/TestFsDatasetImpl/testCleanShutdownOfVolume/
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl.testCleanShutdownOfVolume(TestFsDatasetImpl.java:683)
> Standard Error
> Exception in thread "DataNode:
> [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Java8/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data1/,
>
> [DISK]file:/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Java8/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data2/]]
> heartbeating to localhost/127.0.0.1:35113" java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockReports(FsDatasetImpl.java:1714)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.shutdownBlockPool(FsDatasetImpl.java:2591)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.shutdownBlockPool(DataNode.java:1479)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.shutdownActor(BPOfferService.java:411)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.cleanUp(BPServiceActor.java:494)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:749)
> at java.lang.Thread.run(Thread.java:744)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10260) TestFsDatasetImpl#testCleanShutdownOfVolume often fails
[
https://issues.apache.org/jira/browse/HDFS-10260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264181#comment-15264181
]
Rushabh S Shah commented on HDFS-10260:
---
[~kihwal]: Thanks for reviews and committing the patch.
[~jojochuang]: Thanks for reporting the issue and reviewing the patch.
> TestFsDatasetImpl#testCleanShutdownOfVolume often fails
> ---
>
> Key: HDFS-10260
> URL: https://issues.apache.org/jira/browse/HDFS-10260
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, test
>Reporter: Wei-Chiu Chuang
>Assignee: Rushabh S Shah
> Fix For: 2.8.0
>
> Attachments: HDFS-10260-v1.patch, HDFS-10260.patch
>
>
> This test failure occurs in upstream Jenkins. Looking at the test code, I
> think it should be improved to capture the root cause of failure:
> E.g. change {{Thread.sleep(1000)}} to {{GenericTestUtils.waitFor}} and use
> {{GenericTestUtils.assertExceptionContains}} to replace
> {code}
> Assert.assertTrue(ioe.getMessage().contains(info.toString()));
> {code}
> https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/1062/testReport/junit/org.apache.hadoop.hdfs.server.datanode.fsdataset.impl/TestFsDatasetImpl/testCleanShutdownOfVolume/
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl.testCleanShutdownOfVolume(TestFsDatasetImpl.java:683)
> Standard Error
> Exception in thread "DataNode:
> [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Java8/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data1/,
>
> [DISK]file:/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Java8/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data2/]]
> heartbeating to localhost/127.0.0.1:35113" java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockReports(FsDatasetImpl.java:1714)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.shutdownBlockPool(FsDatasetImpl.java:2591)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.shutdownBlockPool(DataNode.java:1479)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.shutdownActor(BPOfferService.java:411)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.cleanUp(BPServiceActor.java:494)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:749)
> at java.lang.Thread.run(Thread.java:744)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10347) Namenode report bad bad method doesn't log the bad block or datanode.
Rushabh S Shah created HDFS-10347:
-
Summary: Namenode report bad bad method doesn't log the bad block
or datanode.
Key: HDFS-10347
URL: https://issues.apache.org/jira/browse/HDFS-10347
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.7.0
Reporter: Rushabh S Shah
Assignee: Rushabh S Shah
Priority: Minor
Currently the method {{FSNamesystem#reportBadBlocks}} doesn't log any
information regarding the bad block id or the datanode on which corrupt block
is found.
It would be helpful to log that information to debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10347) Namenode report bad bad method doesn't log the bad block or datanode.
[
https://issues.apache.org/jira/browse/HDFS-10347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HDFS-10347:
--
Description:
Currently the method {{FSNamesystem#reportBadBlocks}} doesn't log any
information regarding the bad block id or the datanode on which corrupt block
is detected.
It would be helpful to log that information to debug.
was:
Currently the method {{FSNamesystem#reportBadBlocks}} doesn't log any
information regarding the bad block id or the datanode on which corrupt block
is found.
It would be helpful to log that information to debug.
> Namenode report bad bad method doesn't log the bad block or datanode.
> -
>
> Key: HDFS-10347
> URL: https://issues.apache.org/jira/browse/HDFS-10347
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Minor
>
> Currently the method {{FSNamesystem#reportBadBlocks}} doesn't log any
> information regarding the bad block id or the datanode on which corrupt block
> is detected.
> It would be helpful to log that information to debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10260) TestFsDatasetImpl#testCleanShutdownOfVolume often fails
[
https://issues.apache.org/jira/browse/HDFS-10260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-10260:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
I've committed this to trunk, branch-2 and branch-2.8. Thanks for reporting it
[~jojochuang], and [~shahrs87] for the patch.
> TestFsDatasetImpl#testCleanShutdownOfVolume often fails
> ---
>
> Key: HDFS-10260
> URL: https://issues.apache.org/jira/browse/HDFS-10260
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, test
>Reporter: Wei-Chiu Chuang
>Assignee: Rushabh S Shah
> Fix For: 2.8.0
>
> Attachments: HDFS-10260-v1.patch, HDFS-10260.patch
>
>
> This test failure occurs in upstream Jenkins. Looking at the test code, I
> think it should be improved to capture the root cause of failure:
> E.g. change {{Thread.sleep(1000)}} to {{GenericTestUtils.waitFor}} and use
> {{GenericTestUtils.assertExceptionContains}} to replace
> {code}
> Assert.assertTrue(ioe.getMessage().contains(info.toString()));
> {code}
> https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/1062/testReport/junit/org.apache.hadoop.hdfs.server.datanode.fsdataset.impl/TestFsDatasetImpl/testCleanShutdownOfVolume/
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl.testCleanShutdownOfVolume(TestFsDatasetImpl.java:683)
> Standard Error
> Exception in thread "DataNode:
> [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Java8/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data1/,
>
> [DISK]file:/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Java8/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data2/]]
> heartbeating to localhost/127.0.0.1:35113" java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockReports(FsDatasetImpl.java:1714)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.shutdownBlockPool(FsDatasetImpl.java:2591)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.shutdownBlockPool(DataNode.java:1479)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.shutdownActor(BPOfferService.java:411)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.cleanUp(BPServiceActor.java:494)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:749)
> at java.lang.Thread.run(Thread.java:744)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10260) TestFsDatasetImpl#testCleanShutdownOfVolume often fails
[
https://issues.apache.org/jira/browse/HDFS-10260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-10260:
--
Target Version/s: 2.8.0
> TestFsDatasetImpl#testCleanShutdownOfVolume often fails
> ---
>
> Key: HDFS-10260
> URL: https://issues.apache.org/jira/browse/HDFS-10260
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, test
>Reporter: Wei-Chiu Chuang
>Assignee: Rushabh S Shah
> Attachments: HDFS-10260-v1.patch, HDFS-10260.patch
>
>
> This test failure occurs in upstream Jenkins. Looking at the test code, I
> think it should be improved to capture the root cause of failure:
> E.g. change {{Thread.sleep(1000)}} to {{GenericTestUtils.waitFor}} and use
> {{GenericTestUtils.assertExceptionContains}} to replace
> {code}
> Assert.assertTrue(ioe.getMessage().contains(info.toString()));
> {code}
> https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/1062/testReport/junit/org.apache.hadoop.hdfs.server.datanode.fsdataset.impl/TestFsDatasetImpl/testCleanShutdownOfVolume/
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl.testCleanShutdownOfVolume(TestFsDatasetImpl.java:683)
> Standard Error
> Exception in thread "DataNode:
> [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Java8/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data1/,
>
> [DISK]file:/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Java8/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data2/]]
> heartbeating to localhost/127.0.0.1:35113" java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockReports(FsDatasetImpl.java:1714)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.shutdownBlockPool(FsDatasetImpl.java:2591)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.shutdownBlockPool(DataNode.java:1479)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.shutdownActor(BPOfferService.java:411)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.cleanUp(BPServiceActor.java:494)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:749)
> at java.lang.Thread.run(Thread.java:744)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10260) TestFsDatasetImpl#testCleanShutdownOfVolume often fails
[
https://issues.apache.org/jira/browse/HDFS-10260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264154#comment-15264154
]
Kihwal Lee commented on HDFS-10260:
---
+1 lgtm
> TestFsDatasetImpl#testCleanShutdownOfVolume often fails
> ---
>
> Key: HDFS-10260
> URL: https://issues.apache.org/jira/browse/HDFS-10260
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, test
>Reporter: Wei-Chiu Chuang
>Assignee: Rushabh S Shah
> Attachments: HDFS-10260-v1.patch, HDFS-10260.patch
>
>
> This test failure occurs in upstream Jenkins. Looking at the test code, I
> think it should be improved to capture the root cause of failure:
> E.g. change {{Thread.sleep(1000)}} to {{GenericTestUtils.waitFor}} and use
> {{GenericTestUtils.assertExceptionContains}} to replace
> {code}
> Assert.assertTrue(ioe.getMessage().contains(info.toString()));
> {code}
> https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/1062/testReport/junit/org.apache.hadoop.hdfs.server.datanode.fsdataset.impl/TestFsDatasetImpl/testCleanShutdownOfVolume/
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl.testCleanShutdownOfVolume(TestFsDatasetImpl.java:683)
> Standard Error
> Exception in thread "DataNode:
> [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Java8/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data1/,
>
> [DISK]file:/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Java8/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data2/]]
> heartbeating to localhost/127.0.0.1:35113" java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockReports(FsDatasetImpl.java:1714)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.shutdownBlockPool(FsDatasetImpl.java:2591)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.shutdownBlockPool(DataNode.java:1479)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.shutdownActor(BPOfferService.java:411)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.cleanUp(BPServiceActor.java:494)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:749)
> at java.lang.Thread.run(Thread.java:744)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-334) dfsadmin -metasave should also log corrupt replicas info
[ https://issues.apache.org/jira/browse/HDFS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264012#comment-15264012 ] Denis Bolshakov commented on HDFS-334: -- [~boky01] thanks for assisting, tomorrow I will try to simulate build environment and try again. > dfsadmin -metasave should also log corrupt replicas info > > > Key: HDFS-334 > URL: https://issues.apache.org/jira/browse/HDFS-334 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Lohit Vijayarenu >Assignee: Denis Bolshakov >Priority: Minor > Labels: newbie > Attachments: HDFS-334.001.patch, HDFS-334.002.patch, > HDFS-334.003.patch > > > _hadoop dfsadmin -metasave _ should also dump information about > corrupt replicas map. This could help in telling if pending replication was > due to corrupt replicas. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10276) Different results for exist call for file.ext/name
[
https://issues.apache.org/jira/browse/HDFS-10276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263898#comment-15263898
]
Hadoop QA commented on HDFS-10276:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 32s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
46s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 43s
{color} | {color:green} trunk passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 7s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 25s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 24s
{color} | {color:green} trunk passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 19s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 35s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 11s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 11s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 5s
{color} | {color:red} root: patch generated 1 new + 114 unchanged - 0 fixed =
115 total (was 114) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 23s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 18s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 53s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_91.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 0s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 36s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 42s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 0s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 6s {color}
| {color:red} hadoop-hdfs in the patch failed
[jira] [Commented] (HDFS-334) dfsadmin -metasave should also log corrupt replicas info
[ https://issues.apache.org/jira/browse/HDFS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263757#comment-15263757 ] Andras Bokor commented on HDFS-334: --- [~bolshakov.de...@gmail.com] I think you need to find someone to review your package. Lohit's last action was more than a year ago. > dfsadmin -metasave should also log corrupt replicas info > > > Key: HDFS-334 > URL: https://issues.apache.org/jira/browse/HDFS-334 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Lohit Vijayarenu >Assignee: Denis Bolshakov >Priority: Minor > Labels: newbie > Attachments: HDFS-334.001.patch, HDFS-334.002.patch, > HDFS-334.003.patch > > > _hadoop dfsadmin -metasave _ should also dump information about > corrupt replicas map. This could help in telling if pending replication was > due to corrupt replicas. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10332) hdfs-native-client fails to build with CMake 2.8.11 or earlier
[
https://issues.apache.org/jira/browse/HDFS-10332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263707#comment-15263707
]
Tibor Kiss commented on HDFS-10332:
---
Thanks for committing, James!
> hdfs-native-client fails to build with CMake 2.8.11 or earlier
> --
>
> Key: HDFS-10332
> URL: https://issues.apache.org/jira/browse/HDFS-10332
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Tibor Kiss
>Assignee: Tibor Kiss
>Priority: Minor
> Attachments: HDFS-10332.01.patch, HDFS-10332.HDFS-8707.001.patch
>
>
> Due to a new syntax introduced in CMake 2.8.12 (get_filename_component's
> function when VAR=DIRECTORY) the native-client won't build.
> Currently RHEL6 & 7 are using older version of CMake.
> Error log:
> {noformat}
> [INFO] --- maven-antrun-plugin:1.7:run (make) @ hadoop-hdfs-native-client ---
> [INFO] Executing tasks
> main:
> [exec] JAVA_HOME=,
> JAVA_JVM_LIBRARY=/usr/java/jdk1.7.0_79/jre/lib/amd64/server/libjvm.so
> [exec] JAVA_INCLUDE_PATH=/usr/java/jdk1.7.0_79/include,
> JAVA_INCLUDE_PATH2=/usr/java/jdk1.7.0_79/include/linux
> [exec] Located all JNI components successfully.
> [exec] -- Could NOT find PROTOBUF (missing: PROTOBUF_LIBRARY
> PROTOBUF_INCLUDE_DIR)
> [exec] -- valgrind location: MEMORYCHECK_COMMAND-NOTFOUND
> [exec] -- checking for module 'fuse'
> [exec] -- package 'fuse' not found
> [exec] -- Failed to find Linux FUSE libraries or include files. Will
> not build FUSE client.
> [exec] -- Configuring incomplete, errors occurred!
> [exec] CMake Error at main/native/libhdfspp/CMakeLists.txt:71
> (get_filename_component):
> [exec] get_filename_component unknown component DIRECTORY
> [exec] Call Stack (most recent call first):
> [exec] main/native/libhdfspp/CMakeLists.txt:95 (copy_on_demand)
> [exec]
> [exec]
> [exec] CMake Error at main/native/libhdfspp/CMakeLists.txt:71
> (get_filename_component):
> [exec] get_filename_component unknown component DIRECTORY
> [exec] Call Stack (most recent call first):
> [exec] main/native/libhdfspp/CMakeLists.txt:96 (copy_on_demand)
> [exec]
> [exec]
> [exec] CMake Error at main/native/libhdfspp/CMakeLists.txt:71
> (get_filename_component):
> [exec] get_filename_component unknown component DIRECTORY
> [exec] Call Stack (most recent call first):
> [exec] main/native/libhdfspp/CMakeLists.txt:97 (copy_on_demand)
> [exec]
> [exec]
> [exec] CMake Error at main/native/libhdfspp/CMakeLists.txt:71
> (get_filename_component):
> [exec] get_filename_component unknown component DIRECTORY
> [exec] Call Stack (most recent call first):
> [exec] main/native/libhdfspp/CMakeLists.txt:98 (copy_on_demand)
> [exec]
> [exec]
> [exec] CMake Error: The following variables are used in this project,
> but they are set to NOTFOUND.
> [exec] Please set them or make sure they are set and tested correctly in
> the CMake files:
> [exec] PROTOBUF_LIBRARY (ADVANCED)
> [exec] linked by target "hdfspp" in directory
> /home/tiborkiss/devel/workspace/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp
> [exec] linked by target "hdfspp_static" in directory
> /home/tiborkiss/devel/workspace/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp
> [exec] linked by target "protoc-gen-hrpc" in directory
> /home/tiborkiss/devel/workspace/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/proto
> [exec] linked by target "bad_datanode_test" in directory
> /home/tiborkiss/devel/workspace/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests
> [exec] linked by target "hdfs_builder_test" in directory
> /home/tiborkiss/devel/workspace/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests
> [exec] linked by target "hdfspp_errors_test" in directory
> /home/tiborkiss/devel/workspace
> [exec]
> /hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests
> [exec] linked by target "libhdfs_threaded_hdfspp_test_shim_static"
> in directory
> /home/tiborkiss/devel/workspace/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/test
> [exec] linked by target "logging_test" in directory
> /home/tiborkiss/devel/workspace/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests
> [exec] linked by target "node_exclusion_test" in directory
>
[jira] [Commented] (HDFS-10172) hdfs erasurecode command should remove the redundant -usage option
[
https://issues.apache.org/jira/browse/HDFS-10172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263692#comment-15263692
]
Yuanbo Liu commented on HDFS-10172:
---
The failures are not related to the patch.
[~zhz] please take a look if it's convenient for you. Thanks in advance.
> hdfs erasurecode command should remove the redundant -usage option
> --
>
> Key: HDFS-10172
> URL: https://issues.apache.org/jira/browse/HDFS-10172
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Wei-Chiu Chuang
>Assignee: Yuanbo Liu
>Priority: Minor
> Labels: newbie
> Attachments: HDFS-10172.001.patch, HDFS-10172.002.patch
>
>
> {{hdfs erasurecode}} has two similar options {{-help}} and {{-usage}}.
> {noformat}
> hdfs erasurecode -usage getPolicy
> Usage: hdfs erasurecode [generic options] -getPolicy
> hdfs erasurecode -help getPolicy
> -getPolicy :
> Get erasure coding policy information about at specified path
> {noformat}
> I don't think the -usage option is that useful, because -help option already
> covers that. No other HDFS commands has the same -usage command.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10276) Different results for exist call for file.ext/name
[
https://issues.apache.org/jira/browse/HDFS-10276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuanbo Liu updated HDFS-10276:
--
Status: Patch Available (was: In Progress)
> Different results for exist call for file.ext/name
> --
>
> Key: HDFS-10276
> URL: https://issues.apache.org/jira/browse/HDFS-10276
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kevin Cox
>Assignee: Yuanbo Liu
> Attachments: HDFS-10276.001.patch, HDFS-10276.002.patch,
> HDFS-10276.003.patch, HDFS-10276.004.patch
>
>
> Given you have a file {{/file}} an existence check for the path
> {{/file/whatever}} will give different responses for different
> implementations of FileSystem.
> LocalFileSystem will return false while DistributedFileSystem will throw
> {{org.apache.hadoop.security.AccessControlException: Permission denied: ...,
> access=EXECUTE, ...}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10276) Different results for exist call for file.ext/name
[
https://issues.apache.org/jira/browse/HDFS-10276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuanbo Liu updated HDFS-10276:
--
Attachment: HDFS-10276.004.patch
> Different results for exist call for file.ext/name
> --
>
> Key: HDFS-10276
> URL: https://issues.apache.org/jira/browse/HDFS-10276
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kevin Cox
>Assignee: Yuanbo Liu
> Attachments: HDFS-10276.001.patch, HDFS-10276.002.patch,
> HDFS-10276.003.patch, HDFS-10276.004.patch
>
>
> Given you have a file {{/file}} an existence check for the path
> {{/file/whatever}} will give different responses for different
> implementations of FileSystem.
> LocalFileSystem will return false while DistributedFileSystem will throw
> {{org.apache.hadoop.security.AccessControlException: Permission denied: ...,
> access=EXECUTE, ...}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10276) Different results for exist call for file.ext/name
[
https://issues.apache.org/jira/browse/HDFS-10276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuanbo Liu updated HDFS-10276:
--
Status: In Progress (was: Patch Available)
> Different results for exist call for file.ext/name
> --
>
> Key: HDFS-10276
> URL: https://issues.apache.org/jira/browse/HDFS-10276
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kevin Cox
>Assignee: Yuanbo Liu
> Attachments: HDFS-10276.001.patch, HDFS-10276.002.patch,
> HDFS-10276.003.patch, HDFS-10276.004.patch
>
>
> Given you have a file {{/file}} an existence check for the path
> {{/file/whatever}} will give different responses for different
> implementations of FileSystem.
> LocalFileSystem will return false while DistributedFileSystem will throw
> {{org.apache.hadoop.security.AccessControlException: Permission denied: ...,
> access=EXECUTE, ...}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9758) libhdfs++: Implement Python bindings
[ https://issues.apache.org/jira/browse/HDFS-9758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263651#comment-15263651 ] Tibor Kiss commented on HDFS-9758: -- Thanks for the detailed comments, [~James Clampffer]! I’ve proposed Cython, cpppyy and Boost.Python simply because I thought we’d like to expose the object oriented interface to Python. If that’s not the case CTypes is surely the simplest option without imposing any dependency. Cython would be beneficial if we would either need OO interfaces or we’d prefer performance over programmer productivity. As you already implemented a reasonable amount of code for CTypes I think it is best to extend that further. If you like I can pick up where you left off and finish the remaining calls and include python tests. > libhdfs++: Implement Python bindings > > > Key: HDFS-9758 > URL: https://issues.apache.org/jira/browse/HDFS-9758 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer > Attachments: hdfs_posix.py > > > It'd be really useful to have bindings for various scripting languages. > Python would be a good start because of it's popularity and how easy it is to > interact with shared libraries using the ctypes module. I think bindings for > the V8 engine that nodeJS uses would be a close second in terms of expanding > the potential user base. > Probably worth starting with just adding a synchronous API and building from > there to avoid interactions with python's garbage collector until the > bindings prove to be solid. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10338) DistCp masks potential CRC check failures
[
https://issues.apache.org/jira/browse/HDFS-10338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263646#comment-15263646
]
Hadoop QA commented on HDFS-10338:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
45s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} trunk passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} trunk passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s {color} | {color:green} hadoop-tools/hadoop-distcp: patch generated 0 new +
60 unchanged - 1 fixed = 60 total (was 61) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 58s {color}
| {color:red} hadoop-distcp in the patch failed with JDK v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 45s {color}
| {color:red} hadoop-distcp in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 38s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_91 Failed junit tests | hadoop.tools.TestDistCpViewFs |
| | hadoop.tools.TestIntegration |
| | hadoop.tools.TestExternalCall |
| JDK v1.7.0_95 Failed junit tests | hadoop.tools.TestDistCpViewFs |
| | hadoop.tools.TestIntegration |
| | hadoop.tools.TestExternalCall |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:cf2ee45 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12801393/HDFS-10338.001.patch |
| JIRA Issue | HDFS-10338 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs
[jira] [Assigned] (HDFS-7330) Unclosed RandomAccessFile warnings in FSDatasetIml.
[ https://issues.apache.org/jira/browse/HDFS-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Bokor reassigned HDFS-7330: -- Assignee: Andras Bokor (was: Milan Desai) > Unclosed RandomAccessFile warnings in FSDatasetIml. > --- > > Key: HDFS-7330 > URL: https://issues.apache.org/jira/browse/HDFS-7330 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.5.1 >Reporter: Konstantin Shvachko >Assignee: Andras Bokor > Labels: newbie > > RandomAccessFile is opened as an underline file for FileInputStream. It > should be closed when the stream is closed. So to fix these 2 warning (in > getBlockInputStream() and getTmpInputStreams()) we just need suppress them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10345) [umbrella] Implement an asynchronous DistributedFileSystem
[ https://issues.apache.org/jira/browse/HDFS-10345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263623#comment-15263623 ] Duo Zhang commented on HDFS-10345: -- And I think this will not effect performance since the rpc is born asynchronous... Thanks. > [umbrella] Implement an asynchronous DistributedFileSystem > -- > > Key: HDFS-10345 > URL: https://issues.apache.org/jira/browse/HDFS-10345 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs, hdfs-client >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > > This is proposed to implement an asynchronous DistributedFileSystem based on > AsyncFileSystem APIs in HADOOP-12910. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10345) [umbrella] Implement an asynchronous DistributedFileSystem
[ https://issues.apache.org/jira/browse/HDFS-10345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263614#comment-15263614 ] Duo Zhang commented on HDFS-10345: -- I think it is more reasonable and clean to implement the synchronous DFS based on the asynchronous DFS? Get a Future object by calling the same method in asynchronous DFS, and then call its get method... > [umbrella] Implement an asynchronous DistributedFileSystem > -- > > Key: HDFS-10345 > URL: https://issues.apache.org/jira/browse/HDFS-10345 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs, hdfs-client >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > > This is proposed to implement an asynchronous DistributedFileSystem based on > AsyncFileSystem APIs in HADOOP-12910. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-916) Rewrite DFSOutputStream to use a single thread with NIO
[ https://issues.apache.org/jira/browse/HDFS-916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon reassigned HDFS-916: Assignee: (was: Todd Lipcon) > Rewrite DFSOutputStream to use a single thread with NIO > --- > > Key: HDFS-916 > URL: https://issues.apache.org/jira/browse/HDFS-916 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 0.22.0 >Reporter: Todd Lipcon > > The DFS write pipeline code has some really hairy multi-threaded > synchronization. There have been a lot of bugs produced by this (HDFS-101, > HDFS-793, HDFS-915, tens of others) since it's very hard to understand the > message passing, lock sharing, and interruption properties. The reason for > the multiple threads is to be able to simultaneously send and receive. If > instead of using multiple threads, it used nonblocking IO, I think the whole > thing would be a lot less error prone. > I think we could do this in two halves: one half is the DFSOutputStream. The > other half is BlockReceiver. I opened this JIRA first as I think it's simpler > (only one TCP connection to deal with, rather than an up and downstream) > Opinions? Am I crazy? I would like to see some agreement on the idea before I > spend time writing code. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10346) Implement asynchronous setPermission for DistributedFileSystem
Xiaobing Zhou created HDFS-10346: Summary: Implement asynchronous setPermission for DistributedFileSystem Key: HDFS-10346 URL: https://issues.apache.org/jira/browse/HDFS-10346 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Xiaobing Zhou Assignee: Xiaobing Zhou This is proposed to implement an asynchronous setPermission. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10345) [umbrella] Implement an asynchronous DistributedFileSystem
[ https://issues.apache.org/jira/browse/HDFS-10345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-10345: - Summary: [umbrella] Implement an asynchronous DistributedFileSystem (was: Implement an asynchronous DistributedFileSystem) > [umbrella] Implement an asynchronous DistributedFileSystem > -- > > Key: HDFS-10345 > URL: https://issues.apache.org/jira/browse/HDFS-10345 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs, hdfs-client >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > > This is proposed to implement an asynchronous DistributedFileSystem based on > AsyncFileSystem APIs in HADOOP-12910. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10345) Implement an asynchronous DistributedFileSystem
[ https://issues.apache.org/jira/browse/HDFS-10345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-10345: - Component/s: hdfs-client hdfs > Implement an asynchronous DistributedFileSystem > --- > > Key: HDFS-10345 > URL: https://issues.apache.org/jira/browse/HDFS-10345 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs, hdfs-client >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > > This is proposed to implement an asynchronous DistributedFileSystem based on > AsyncFileSystem APIs in HADOOP-12910. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10345) Implement an asynchronous DistributedFileSystem
Xiaobing Zhou created HDFS-10345: Summary: Implement an asynchronous DistributedFileSystem Key: HDFS-10345 URL: https://issues.apache.org/jira/browse/HDFS-10345 Project: Hadoop HDFS Issue Type: New Feature Reporter: Xiaobing Zhou Assignee: Xiaobing Zhou This is proposed to implement an asynchronous DistributedFileSystem based on AsyncFileSystem APIs in HADOOP-12910. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-223) Asynchronous IO Handling in Hadoop and HDFS
[
https://issues.apache.org/jira/browse/HDFS-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263601#comment-15263601
]
Duo Zhang commented on HDFS-223:
In HBASE-14790 we have implemented a {{FanOutOneBlockAsyncDFSOutput}} based on
netty. It performs much better than the default {{DFSOutputStream}} for WAL in
HBase.
And we plan to move these stuffs into HDFS since it should belong to HDFS. Of
course this is not a simple copy-paste work, the implementation in HBase is not
suitable for general use.
Will come back later with a proposal on the asynchronous API first.
Thanks.
> Asynchronous IO Handling in Hadoop and HDFS
> ---
>
> Key: HDFS-223
> URL: https://issues.apache.org/jira/browse/HDFS-223
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Raghu Angadi
> Attachments: GrizzlyEchoServer.patch, MinaEchoServer.patch
>
>
> I think Hadoop needs utilities or framework to make it simpler to deal with
> generic asynchronous IO in Hadoop.
> Example use case :
> Its been a long standing problem that DataNode takes too many threads for
> data transfers. Each write operation takes up 2 threads at each of the
> datanodes and each read operation takes one irrespective of how much activity
> is on the sockets. The kinds of load that HDFS serves has been expanding
> quite fast and HDFS should handle these varied loads better. If there is a
> framework for non-blocking IO, read and write pipeline state machines could
> be implemented with async events on a fixed number of threads.
> A generic utility is better since it could be used in other places like
> DFSClient. DFSClient currently creates 2 extra threads for each file it has
> open for writing.
> Initially I started writing a primitive "selector", then tried to see if such
> facility already exists. [Apache MINA|http://mina.apache.org] seemed to do
> exactly this. My impression after looking the the interface and examples is
> that it does not give kind control we might prefer or need. First use case I
> was thinking of implementing using MINA was to replace "response handlers" in
> DataNode. The response handlers are simpler since they don't involve disk
> I/O. I [asked on MINA user
> list|http://www.nabble.com/Async-events-with-existing-NIO-sockets.-td18640767.html],
> but looks like it can not be done, I think mainly because the sockets are
> already created.
> Essentially what I have in mind is similar to MINA, except that read and
> write of the sockets is done by the event handlers. The lowest layer
> essentially invokes selectors, invokes event handlers on single or on
> multiple threads. Each event handler is is expected to do some non-blocking
> work. We would of course have utility handler implementations that do read,
> write, accept etc, that are useful for simple processing.
> Sam Pullara mentioned that [xSockets|http://xsocket.sourceforge.net/] is more
> flexible. It is under GPL.
> Are there other such implementations we should look at?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10224) Implement asynchronous rename for DistributedFileSystem
[ https://issues.apache.org/jira/browse/HDFS-10224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-10224: - Issue Type: Sub-task (was: New Feature) Parent: HDFS-10345 > Implement asynchronous rename for DistributedFileSystem > --- > > Key: HDFS-10224 > URL: https://issues.apache.org/jira/browse/HDFS-10224 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs, hdfs-client >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Fix For: 2.8.0 > > Attachments: HDFS-10224-HDFS-9924.000.patch, > HDFS-10224-HDFS-9924.001.patch, HDFS-10224-HDFS-9924.002.patch, > HDFS-10224-HDFS-9924.003.patch, HDFS-10224-HDFS-9924.004.patch, > HDFS-10224-HDFS-9924.005.patch, HDFS-10224-HDFS-9924.006.patch, > HDFS-10224-HDFS-9924.007.patch, HDFS-10224-HDFS-9924.008.patch, > HDFS-10224-HDFS-9924.009.patch, HDFS-10224-and-HADOOP-12909.000.patch > > > This is proposed to implement an asynchronous rename. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10224) Implement asynchronous rename for DistributedFileSystem
[ https://issues.apache.org/jira/browse/HDFS-10224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-10224: - Issue Type: New Feature (was: Sub-task) Parent: (was: HDFS-9924) > Implement asynchronous rename for DistributedFileSystem > --- > > Key: HDFS-10224 > URL: https://issues.apache.org/jira/browse/HDFS-10224 > Project: Hadoop HDFS > Issue Type: New Feature > Components: fs, hdfs-client >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Fix For: 2.8.0 > > Attachments: HDFS-10224-HDFS-9924.000.patch, > HDFS-10224-HDFS-9924.001.patch, HDFS-10224-HDFS-9924.002.patch, > HDFS-10224-HDFS-9924.003.patch, HDFS-10224-HDFS-9924.004.patch, > HDFS-10224-HDFS-9924.005.patch, HDFS-10224-HDFS-9924.006.patch, > HDFS-10224-HDFS-9924.007.patch, HDFS-10224-HDFS-9924.008.patch, > HDFS-10224-HDFS-9924.009.patch, HDFS-10224-and-HADOOP-12909.000.patch > > > This is proposed to implement an asynchronous rename. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10224) Implement asynchronous rename for DistributedFileSystem
[ https://issues.apache.org/jira/browse/HDFS-10224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-10224: - Description: This is proposed to implement an asynchronous rename. (was: This is proposed to implement an asynchronous DistributedFileSystem based on AsyncFileSystem APIs in HADOOP-12910. In addition, rename is implemented in this JIRA.) > Implement asynchronous rename for DistributedFileSystem > --- > > Key: HDFS-10224 > URL: https://issues.apache.org/jira/browse/HDFS-10224 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs, hdfs-client >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Fix For: 2.8.0 > > Attachments: HDFS-10224-HDFS-9924.000.patch, > HDFS-10224-HDFS-9924.001.patch, HDFS-10224-HDFS-9924.002.patch, > HDFS-10224-HDFS-9924.003.patch, HDFS-10224-HDFS-9924.004.patch, > HDFS-10224-HDFS-9924.005.patch, HDFS-10224-HDFS-9924.006.patch, > HDFS-10224-HDFS-9924.007.patch, HDFS-10224-HDFS-9924.008.patch, > HDFS-10224-HDFS-9924.009.patch, HDFS-10224-and-HADOOP-12909.000.patch > > > This is proposed to implement an asynchronous rename. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
