[jira] [Updated] (HDFS-4448) HA NN does not start with wildcard address configured for other NN when security is enabled
[
https://issues.apache.org/jira/browse/HDFS-4448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HDFS-4448:
-
Attachment: HDFS-4448.patch
Whoops, left an unused variable in the previous patch. This patch is to address
that findbugs warning.
> HA NN does not start with wildcard address configured for other NN when
> security is enabled
> ---
>
> Key: HDFS-4448
> URL: https://issues.apache.org/jira/browse/HDFS-4448
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, namenode, security
>Affects Versions: 2.0.3-alpha
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
> Attachments: HDFS-4448.patch, HDFS-4448.patch
>
>
> Currently if one tries to configure HA NNs use the wildcard HTTP address when
> security is enabled, the NN will fail to start with an error like the
> following:
> {code}
> java.lang.IllegalArgumentException: java.io.IOException: Cannot use a
> wildcard address with security. Must explicitly set bind address for Kerberos
> {code}
> This is the case even if one configures an actual address for the other NN's
> HTTP address. There's no good reason for this, since we now check for the
> local address being set to 0.0.0.0 and determine the canonical hostname for
> Kerberos purposes using
> {{InetAddress.getLocalHost().getCanonicalHostName()}}, so we should remove
> the restriction.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4448) HA NN does not start with wildcard address configured for other NN when security is enabled
[
https://issues.apache.org/jira/browse/HDFS-4448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565110#comment-13565110
]
Hadoop QA commented on HDFS-4448:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12566904/HDFS-4448.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/3901//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/3901//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3901//console
This message is automatically generated.
> HA NN does not start with wildcard address configured for other NN when
> security is enabled
> ---
>
> Key: HDFS-4448
> URL: https://issues.apache.org/jira/browse/HDFS-4448
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, namenode, security
>Affects Versions: 2.0.3-alpha
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
> Attachments: HDFS-4448.patch
>
>
> Currently if one tries to configure HA NNs use the wildcard HTTP address when
> security is enabled, the NN will fail to start with an error like the
> following:
> {code}
> java.lang.IllegalArgumentException: java.io.IOException: Cannot use a
> wildcard address with security. Must explicitly set bind address for Kerberos
> {code}
> This is the case even if one configures an actual address for the other NN's
> HTTP address. There's no good reason for this, since we now check for the
> local address being set to 0.0.0.0 and determine the canonical hostname for
> Kerberos purposes using
> {{InetAddress.getLocalHost().getCanonicalHostName()}}, so we should remove
> the restriction.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4449) When a decommission is awaiting closure of live blocks, show the block IDs on the NameNode's UI report
Harsh J created HDFS-4449: - Summary: When a decommission is awaiting closure of live blocks, show the block IDs on the NameNode's UI report Key: HDFS-4449 URL: https://issues.apache.org/jira/browse/HDFS-4449 Project: Hadoop HDFS Issue Type: Improvement Reporter: Harsh J Assignee: Harsh J It is rather common for people to be complaining about 'DN decommission' hangs cause of live blocks waiting to get completed by some app (especially certain HBase specifics cause a file to be open for a longer time, as compared with MR/etc.). While they can see a count of the blocks that are live, we should add some more details to that view. Particularly add the list of live blocks waiting to be closed, so that a user may understand better on why its hung and also be able to trace back the block to files manually if needed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3598) WebHDFS: support file concat
[
https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565086#comment-13565086
]
Hadoop QA commented on HDFS-3598:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12566898/HDFS-3598.trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/3899//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/3899//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3899//console
This message is automatically generated.
> WebHDFS: support file concat
>
>
> Key: HDFS-3598
> URL: https://issues.apache.org/jira/browse/HDFS-3598
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.0.3-alpha
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Plamen Jeliazkov
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: HDFS-3598.trunk.patch, HDFS-3598.trunk.patch
>
>
> In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path
> [] psrcs) method. WebHDFS should support it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4448) HA NN does not start with wildcard address configured for other NN when security is enabled
[
https://issues.apache.org/jira/browse/HDFS-4448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HDFS-4448:
-
Attachment: HDFS-4448.patch
Here's a patch which addresses the issue by simply removing the check which is
now overly-restrictive.
No tests are included since to test this adequately one needs multiple hosts
and security to be enabled. I tested this patch on a secure 2-node HA cluster
where each NN is configured itself to bind to 0.0.0.0, but is configured with
an actual address for the other node. I confirmed that everything started up
and checkpointing works as expected.
> HA NN does not start with wildcard address configured for other NN when
> security is enabled
> ---
>
> Key: HDFS-4448
> URL: https://issues.apache.org/jira/browse/HDFS-4448
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, namenode, security
>Affects Versions: 2.0.3-alpha
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
> Attachments: HDFS-4448.patch
>
>
> Currently if one tries to configure HA NNs use the wildcard HTTP address when
> security is enabled, the NN will fail to start with an error like the
> following:
> {code}
> java.lang.IllegalArgumentException: java.io.IOException: Cannot use a
> wildcard address with security. Must explicitly set bind address for Kerberos
> {code}
> This is the case even if one configures an actual address for the other NN's
> HTTP address. There's no good reason for this, since we now check for the
> local address being set to 0.0.0.0 and determine the canonical hostname for
> Kerberos purposes using
> {{InetAddress.getLocalHost().getCanonicalHostName()}}, so we should remove
> the restriction.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4448) HA NN does not start with wildcard address configured for other NN when security is enabled
[
https://issues.apache.org/jira/browse/HDFS-4448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HDFS-4448:
-
Status: Patch Available (was: Open)
> HA NN does not start with wildcard address configured for other NN when
> security is enabled
> ---
>
> Key: HDFS-4448
> URL: https://issues.apache.org/jira/browse/HDFS-4448
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, namenode, security
>Affects Versions: 2.0.3-alpha
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
> Attachments: HDFS-4448.patch
>
>
> Currently if one tries to configure HA NNs use the wildcard HTTP address when
> security is enabled, the NN will fail to start with an error like the
> following:
> {code}
> java.lang.IllegalArgumentException: java.io.IOException: Cannot use a
> wildcard address with security. Must explicitly set bind address for Kerberos
> {code}
> This is the case even if one configures an actual address for the other NN's
> HTTP address. There's no good reason for this, since we now check for the
> local address being set to 0.0.0.0 and determine the canonical hostname for
> Kerberos purposes using
> {{InetAddress.getLocalHost().getCanonicalHostName()}}, so we should remove
> the restriction.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3598) WebHDFS: support file concat
[
https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565058#comment-13565058
]
Hadoop QA commented on HDFS-3598:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12566887/HDFS-3598.trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.fs.TestFilterFileSystem
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/3897//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/3897//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3897//console
This message is automatically generated.
> WebHDFS: support file concat
>
>
> Key: HDFS-3598
> URL: https://issues.apache.org/jira/browse/HDFS-3598
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.0.3-alpha
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Plamen Jeliazkov
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: HDFS-3598.trunk.patch, HDFS-3598.trunk.patch
>
>
> In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path
> [] psrcs) method. WebHDFS should support it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4448) HA NN does not start with wildcard address configured for other NN when security is enabled
Aaron T. Myers created HDFS-4448:
Summary: HA NN does not start with wildcard address configured for
other NN when security is enabled
Key: HDFS-4448
URL: https://issues.apache.org/jira/browse/HDFS-4448
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, namenode, security
Affects Versions: 2.0.3-alpha
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Currently if one tries to configure HA NNs use the wildcard HTTP address when
security is enabled, the NN will fail to start with an error like the following:
{code}
java.lang.IllegalArgumentException: java.io.IOException: Cannot use a wildcard
address with security. Must explicitly set bind address for Kerberos
{code}
This is the case even if one configures an actual address for the other NN's
HTTP address. There's no good reason for this, since we now check for the local
address being set to 0.0.0.0 and determine the canonical hostname for Kerberos
purposes using {{InetAddress.getLocalHost().getCanonicalHostName()}}, so we
should remove the restriction.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-347) DFS read performance suboptimal when client co-located on nodes with data
[
https://issues.apache.org/jira/browse/HDFS-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565055#comment-13565055
]
Hadoop QA commented on HDFS-347:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12566903/2013.01.28.design.pdf
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3900//console
This message is automatically generated.
> DFS read performance suboptimal when client co-located on nodes with data
> -
>
> Key: HDFS-347
> URL: https://issues.apache.org/jira/browse/HDFS-347
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, hdfs-client, performance
>Reporter: George Porter
>Assignee: Colin Patrick McCabe
> Attachments: 2013.01.28.design.pdf, all.tsv, BlockReaderLocal1.txt,
> full.patch, HADOOP-4801.1.patch, HADOOP-4801.2.patch, HADOOP-4801.3.patch,
> HDFS-347-016_cleaned.patch, HDFS-347.016.patch, HDFS-347.017.clean.patch,
> HDFS-347.017.patch, HDFS-347.018.clean.patch, HDFS-347.018.patch2,
> HDFS-347.019.patch, HDFS-347.020.patch, HDFS-347.021.patch,
> HDFS-347.022.patch, HDFS-347.024.patch, HDFS-347.025.patch,
> HDFS-347.026.patch, HDFS-347.027.patch, HDFS-347.029.patch,
> HDFS-347.030.patch, HDFS-347.033.patch, HDFS-347.035.patch,
> HDFS-347-branch-20-append.txt, hdfs-347-merge.txt, hdfs-347-merge.txt,
> hdfs-347-merge.txt, hdfs-347.png, hdfs-347.txt, local-reads-doc
>
>
> One of the major strategies Hadoop uses to get scalable data processing is to
> move the code to the data. However, putting the DFS client on the same
> physical node as the data blocks it acts on doesn't improve read performance
> as much as expected.
> After looking at Hadoop and O/S traces (via HADOOP-4049), I think the problem
> is due to the HDFS streaming protocol causing many more read I/O operations
> (iops) than necessary. Consider the case of a DFSClient fetching a 64 MB
> disk block from the DataNode process (running in a separate JVM) running on
> the same machine. The DataNode will satisfy the single disk block request by
> sending data back to the HDFS client in 64-KB chunks. In BlockSender.java,
> this is done in the sendChunk() method, relying on Java's transferTo()
> method. Depending on the host O/S and JVM implementation, transferTo() is
> implemented as either a sendfilev() syscall or a pair of mmap() and write().
> In either case, each chunk is read from the disk by issuing a separate I/O
> operation for each chunk. The result is that the single request for a 64-MB
> block ends up hitting the disk as over a thousand smaller requests for 64-KB
> each.
> Since the DFSClient runs in a different JVM and process than the DataNode,
> shuttling data from the disk to the DFSClient also results in context
> switches each time network packets get sent (in this case, the 64-kb chunk
> turns into a large number of 1500 byte packet send operations). Thus we see
> a large number of context switches for each block send operation.
> I'd like to get some feedback on the best way to address this, but I think
> providing a mechanism for a DFSClient to directly open data blocks that
> happen to be on the same machine. It could do this by examining the set of
> LocatedBlocks returned by the NameNode, marking those that should be resident
> on the local host. Since the DataNode and DFSClient (probably) share the
> same hadoop configuration, the DFSClient should be able to find the files
> holding the block data, and it could directly open them and send data back to
> the client. This would avoid the context switches imposed by the network
> layer, and would allow for much larger read buffers than 64KB, which should
> reduce the number of iops imposed by each read block operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-347) DFS read performance suboptimal when client co-located on nodes with data
[ https://issues.apache.org/jira/browse/HDFS-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565052#comment-13565052 ] Colin Patrick McCabe commented on HDFS-347: --- bq. In the code above, when would "DomainSocket#anchorNative got error: unknown" be used? it's not used; this assignment could be removed. > DFS read performance suboptimal when client co-located on nodes with data > - > > Key: HDFS-347 > URL: https://issues.apache.org/jira/browse/HDFS-347 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, hdfs-client, performance >Reporter: George Porter >Assignee: Colin Patrick McCabe > Attachments: 2013.01.28.design.pdf, all.tsv, BlockReaderLocal1.txt, > full.patch, HADOOP-4801.1.patch, HADOOP-4801.2.patch, HADOOP-4801.3.patch, > HDFS-347-016_cleaned.patch, HDFS-347.016.patch, HDFS-347.017.clean.patch, > HDFS-347.017.patch, HDFS-347.018.clean.patch, HDFS-347.018.patch2, > HDFS-347.019.patch, HDFS-347.020.patch, HDFS-347.021.patch, > HDFS-347.022.patch, HDFS-347.024.patch, HDFS-347.025.patch, > HDFS-347.026.patch, HDFS-347.027.patch, HDFS-347.029.patch, > HDFS-347.030.patch, HDFS-347.033.patch, HDFS-347.035.patch, > HDFS-347-branch-20-append.txt, hdfs-347-merge.txt, hdfs-347-merge.txt, > hdfs-347-merge.txt, hdfs-347.png, hdfs-347.txt, local-reads-doc > > > One of the major strategies Hadoop uses to get scalable data processing is to > move the code to the data. However, putting the DFS client on the same > physical node as the data blocks it acts on doesn't improve read performance > as much as expected. > After looking at Hadoop and O/S traces (via HADOOP-4049), I think the problem > is due to the HDFS streaming protocol causing many more read I/O operations > (iops) than necessary. Consider the case of a DFSClient fetching a 64 MB > disk block from the DataNode process (running in a separate JVM) running on > the same machine. The DataNode will satisfy the single disk block request by > sending data back to the HDFS client in 64-KB chunks. In BlockSender.java, > this is done in the sendChunk() method, relying on Java's transferTo() > method. Depending on the host O/S and JVM implementation, transferTo() is > implemented as either a sendfilev() syscall or a pair of mmap() and write(). > In either case, each chunk is read from the disk by issuing a separate I/O > operation for each chunk. The result is that the single request for a 64-MB > block ends up hitting the disk as over a thousand smaller requests for 64-KB > each. > Since the DFSClient runs in a different JVM and process than the DataNode, > shuttling data from the disk to the DFSClient also results in context > switches each time network packets get sent (in this case, the 64-kb chunk > turns into a large number of 1500 byte packet send operations). Thus we see > a large number of context switches for each block send operation. > I'd like to get some feedback on the best way to address this, but I think > providing a mechanism for a DFSClient to directly open data blocks that > happen to be on the same machine. It could do this by examining the set of > LocatedBlocks returned by the NameNode, marking those that should be resident > on the local host. Since the DataNode and DFSClient (probably) share the > same hadoop configuration, the DFSClient should be able to find the files > holding the block data, and it could directly open them and send data back to > the client. This would avoid the context switches imposed by the network > layer, and would allow for much larger read buffers than 64KB, which should > reduce the number of iops imposed by each read block operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-347) DFS read performance suboptimal when client co-located on nodes with data
[ https://issues.apache.org/jira/browse/HDFS-347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-347: -- Attachment: 2013.01.28.design.pdf update design document > DFS read performance suboptimal when client co-located on nodes with data > - > > Key: HDFS-347 > URL: https://issues.apache.org/jira/browse/HDFS-347 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, hdfs-client, performance >Reporter: George Porter >Assignee: Colin Patrick McCabe > Attachments: 2013.01.28.design.pdf, all.tsv, BlockReaderLocal1.txt, > full.patch, HADOOP-4801.1.patch, HADOOP-4801.2.patch, HADOOP-4801.3.patch, > HDFS-347-016_cleaned.patch, HDFS-347.016.patch, HDFS-347.017.clean.patch, > HDFS-347.017.patch, HDFS-347.018.clean.patch, HDFS-347.018.patch2, > HDFS-347.019.patch, HDFS-347.020.patch, HDFS-347.021.patch, > HDFS-347.022.patch, HDFS-347.024.patch, HDFS-347.025.patch, > HDFS-347.026.patch, HDFS-347.027.patch, HDFS-347.029.patch, > HDFS-347.030.patch, HDFS-347.033.patch, HDFS-347.035.patch, > HDFS-347-branch-20-append.txt, hdfs-347-merge.txt, hdfs-347-merge.txt, > hdfs-347-merge.txt, hdfs-347.png, hdfs-347.txt, local-reads-doc > > > One of the major strategies Hadoop uses to get scalable data processing is to > move the code to the data. However, putting the DFS client on the same > physical node as the data blocks it acts on doesn't improve read performance > as much as expected. > After looking at Hadoop and O/S traces (via HADOOP-4049), I think the problem > is due to the HDFS streaming protocol causing many more read I/O operations > (iops) than necessary. Consider the case of a DFSClient fetching a 64 MB > disk block from the DataNode process (running in a separate JVM) running on > the same machine. The DataNode will satisfy the single disk block request by > sending data back to the HDFS client in 64-KB chunks. In BlockSender.java, > this is done in the sendChunk() method, relying on Java's transferTo() > method. Depending on the host O/S and JVM implementation, transferTo() is > implemented as either a sendfilev() syscall or a pair of mmap() and write(). > In either case, each chunk is read from the disk by issuing a separate I/O > operation for each chunk. The result is that the single request for a 64-MB > block ends up hitting the disk as over a thousand smaller requests for 64-KB > each. > Since the DFSClient runs in a different JVM and process than the DataNode, > shuttling data from the disk to the DFSClient also results in context > switches each time network packets get sent (in this case, the 64-kb chunk > turns into a large number of 1500 byte packet send operations). Thus we see > a large number of context switches for each block send operation. > I'd like to get some feedback on the best way to address this, but I think > providing a mechanism for a DFSClient to directly open data blocks that > happen to be on the same machine. It could do this by examining the set of > LocatedBlocks returned by the NameNode, marking those that should be resident > on the local host. Since the DataNode and DFSClient (probably) share the > same hadoop configuration, the DFSClient should be able to find the files > holding the block data, and it could directly open them and send data back to > the client. This would avoid the context switches imposed by the network > layer, and would allow for much larger read buffers than 64KB, which should > reduce the number of iops imposed by each read block operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4424) fsdataset Mkdirs failed cause nullpointexception and other bad consequence
[
https://issues.apache.org/jira/browse/HDFS-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565039#comment-13565039
]
Li Junjun commented on HDFS-4424:
-
this patch for branch1 . why qa apply it to hadoop v2 ?
> fsdataset Mkdirs failed cause nullpointexception and other bad
> consequence
> ---
>
> Key: HDFS-4424
> URL: https://issues.apache.org/jira/browse/HDFS-4424
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 1.0.1
>Reporter: Li Junjun
> Attachments: patch.txt
>
>
> File: /hadoop-1.0.1/hdfs/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
> from line 205:
> {code}
> if (children == null || children.length == 0) {
> children = new FSDir[maxBlocksPerDir];
> for (int idx = 0; idx < maxBlocksPerDir; idx++) {
> children[idx] = new FSDir(new File(dir,
> DataStorage.BLOCK_SUBDIR_PREFIX+idx));
> }
> }
> {code}
> in FSDir constructer method if faild ( space full,so mkdir fails), but
> the children still in use !
> the the write comes(after I run balancer ) , when choose FSDir
> line 192:
> File file = children[idx].addBlock(b, src, false, resetIdx);
> cause exceptions like this
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSDir.addBlock(FSDataset.java:192)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSDir.addBlock(FSDataset.java:192)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSDir.addBlock(FSDataset.java:158)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.addBlock(FSDataset.java:495)
> {code}
>
> should it like this
> {code}
> if (children == null || children.length == 0) {
> List childrenList = new ArrayList();
>
> for (int idx = 0; idx < maxBlocksPerDir; idx++) {
> try{
>childrenList .add( new FSDir(new File(dir,
> DataStorage.BLOCK_SUBDIR_PREFIX+idx)));
> }catch(Exception e){
> }
> children = childrenList.toArray();
> }
> }
> {code}
>
> bad consequence , in my cluster ,this datanode's num blocks became 0 .
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-347) DFS read performance suboptimal when client co-located on nodes with data
[ https://issues.apache.org/jira/browse/HDFS-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565036#comment-13565036 ] Tsz Wo (Nicholas), SZE commented on HDFS-347: - It looks that the patch has a lot of unrelated code/changes. It seems that the branch has not merged with the latest trunk. > DFS read performance suboptimal when client co-located on nodes with data > - > > Key: HDFS-347 > URL: https://issues.apache.org/jira/browse/HDFS-347 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, hdfs-client, performance >Reporter: George Porter >Assignee: Colin Patrick McCabe > Attachments: all.tsv, BlockReaderLocal1.txt, full.patch, > HADOOP-4801.1.patch, HADOOP-4801.2.patch, HADOOP-4801.3.patch, > HDFS-347-016_cleaned.patch, HDFS-347.016.patch, HDFS-347.017.clean.patch, > HDFS-347.017.patch, HDFS-347.018.clean.patch, HDFS-347.018.patch2, > HDFS-347.019.patch, HDFS-347.020.patch, HDFS-347.021.patch, > HDFS-347.022.patch, HDFS-347.024.patch, HDFS-347.025.patch, > HDFS-347.026.patch, HDFS-347.027.patch, HDFS-347.029.patch, > HDFS-347.030.patch, HDFS-347.033.patch, HDFS-347.035.patch, > HDFS-347-branch-20-append.txt, hdfs-347-merge.txt, hdfs-347-merge.txt, > hdfs-347-merge.txt, hdfs-347.png, hdfs-347.txt, local-reads-doc > > > One of the major strategies Hadoop uses to get scalable data processing is to > move the code to the data. However, putting the DFS client on the same > physical node as the data blocks it acts on doesn't improve read performance > as much as expected. > After looking at Hadoop and O/S traces (via HADOOP-4049), I think the problem > is due to the HDFS streaming protocol causing many more read I/O operations > (iops) than necessary. Consider the case of a DFSClient fetching a 64 MB > disk block from the DataNode process (running in a separate JVM) running on > the same machine. The DataNode will satisfy the single disk block request by > sending data back to the HDFS client in 64-KB chunks. In BlockSender.java, > this is done in the sendChunk() method, relying on Java's transferTo() > method. Depending on the host O/S and JVM implementation, transferTo() is > implemented as either a sendfilev() syscall or a pair of mmap() and write(). > In either case, each chunk is read from the disk by issuing a separate I/O > operation for each chunk. The result is that the single request for a 64-MB > block ends up hitting the disk as over a thousand smaller requests for 64-KB > each. > Since the DFSClient runs in a different JVM and process than the DataNode, > shuttling data from the disk to the DFSClient also results in context > switches each time network packets get sent (in this case, the 64-kb chunk > turns into a large number of 1500 byte packet send operations). Thus we see > a large number of context switches for each block send operation. > I'd like to get some feedback on the best way to address this, but I think > providing a mechanism for a DFSClient to directly open data blocks that > happen to be on the same machine. It could do this by examining the set of > LocatedBlocks returned by the NameNode, marking those that should be resident > on the local host. Since the DataNode and DFSClient (probably) share the > same hadoop configuration, the DFSClient should be able to find the files > holding the block data, and it could directly open them and send data back to > the client. This would avoid the context switches imposed by the network > layer, and would allow for much larger read buffers than 64KB, which should > reduce the number of iops imposed by each read block operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4424) fsdataset Mkdirs failed cause nullpointexception and other bad consequence
[
https://issues.apache.org/jira/browse/HDFS-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565034#comment-13565034
]
Hadoop QA commented on HDFS-4424:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12566894/patch.txt
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3898//console
This message is automatically generated.
> fsdataset Mkdirs failed cause nullpointexception and other bad
> consequence
> ---
>
> Key: HDFS-4424
> URL: https://issues.apache.org/jira/browse/HDFS-4424
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 1.0.1
>Reporter: Li Junjun
> Attachments: patch.txt
>
>
> File: /hadoop-1.0.1/hdfs/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
> from line 205:
> {code}
> if (children == null || children.length == 0) {
> children = new FSDir[maxBlocksPerDir];
> for (int idx = 0; idx < maxBlocksPerDir; idx++) {
> children[idx] = new FSDir(new File(dir,
> DataStorage.BLOCK_SUBDIR_PREFIX+idx));
> }
> }
> {code}
> in FSDir constructer method if faild ( space full,so mkdir fails), but
> the children still in use !
> the the write comes(after I run balancer ) , when choose FSDir
> line 192:
> File file = children[idx].addBlock(b, src, false, resetIdx);
> cause exceptions like this
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSDir.addBlock(FSDataset.java:192)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSDir.addBlock(FSDataset.java:192)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSDir.addBlock(FSDataset.java:158)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.addBlock(FSDataset.java:495)
> {code}
>
> should it like this
> {code}
> if (children == null || children.length == 0) {
> List childrenList = new ArrayList();
>
> for (int idx = 0; idx < maxBlocksPerDir; idx++) {
> try{
>childrenList .add( new FSDir(new File(dir,
> DataStorage.BLOCK_SUBDIR_PREFIX+idx)));
> }catch(Exception e){
> }
> children = childrenList.toArray();
> }
> }
> {code}
>
> bad consequence , in my cluster ,this datanode's num blocks became 0 .
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Status: Patch Available (was: Open) > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.3-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.3-alpha > > Attachments: HDFS-3598.trunk.patch, HDFS-3598.trunk.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565032#comment-13565032 ] Plamen Jeliazkov commented on HDFS-3598: I apologize for the patch spam. I was getting a mismatch between trunk and 2.0.3 and was confused for a long time as to why the patch would not apply on Jenkins. I have gotten it correct. Latest patch fixes an issue with FilterFileSystem also requiring concat due to TestFilterFileSystem exclusively checking the implementation to match FileSystem's methods. > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.3-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.3-alpha > > Attachments: HDFS-3598.trunk.patch, HDFS-3598.trunk.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Status: Open (was: Patch Available) > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.3-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.3-alpha > > Attachments: HDFS-3598.trunk.patch, HDFS-3598.trunk.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4424) fsdataset Mkdirs failed cause nullpointexception and other bad consequence
[
https://issues.apache.org/jira/browse/HDFS-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Junjun updated HDFS-4424:
Status: Patch Available (was: Open)
> fsdataset Mkdirs failed cause nullpointexception and other bad
> consequence
> ---
>
> Key: HDFS-4424
> URL: https://issues.apache.org/jira/browse/HDFS-4424
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 1.0.1
>Reporter: Li Junjun
> Attachments: patch.txt
>
>
> File: /hadoop-1.0.1/hdfs/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
> from line 205:
> {code}
> if (children == null || children.length == 0) {
> children = new FSDir[maxBlocksPerDir];
> for (int idx = 0; idx < maxBlocksPerDir; idx++) {
> children[idx] = new FSDir(new File(dir,
> DataStorage.BLOCK_SUBDIR_PREFIX+idx));
> }
> }
> {code}
> in FSDir constructer method if faild ( space full,so mkdir fails), but
> the children still in use !
> the the write comes(after I run balancer ) , when choose FSDir
> line 192:
> File file = children[idx].addBlock(b, src, false, resetIdx);
> cause exceptions like this
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSDir.addBlock(FSDataset.java:192)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSDir.addBlock(FSDataset.java:192)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSDir.addBlock(FSDataset.java:158)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.addBlock(FSDataset.java:495)
> {code}
>
> should it like this
> {code}
> if (children == null || children.length == 0) {
> List childrenList = new ArrayList();
>
> for (int idx = 0; idx < maxBlocksPerDir; idx++) {
> try{
>childrenList .add( new FSDir(new File(dir,
> DataStorage.BLOCK_SUBDIR_PREFIX+idx)));
> }catch(Exception e){
> }
> children = childrenList.toArray();
> }
> }
> {code}
>
> bad consequence , in my cluster ,this datanode's num blocks became 0 .
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Attachment: HDFS-3598.trunk.patch > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.3-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.3-alpha > > Attachments: HDFS-3598.trunk.patch, HDFS-3598.trunk.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4131) Add a tool to print the diff between two snapshots and diff of a snapshot from the current tree
[ https://issues.apache.org/jira/browse/HDFS-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4131: Attachment: HDFS-4131.005.patch Update the patch. Do not distinguish earlier/later snapshots in FSNamesystem#getSnapshotDiffReport. Instead, only compute diff between source and target snapshot where source may be taken after target. > Add a tool to print the diff between two snapshots and diff of a snapshot > from the current tree > --- > > Key: HDFS-4131 > URL: https://issues.apache.org/jira/browse/HDFS-4131 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: Snapshot (HDFS-2802) >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Attachments: HDFS-4131.001.patch, HDFS-4131.002.patch, > HDFS-4131.003.patch, HDFS-4131.004.patch, HDFS-4131.005.patch > > > This jira provides internal data structures and computation processes for > calculating and representing the diff between two snapshots, or the diff > between a snapshot and the current tree. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4424) fsdataset Mkdirs failed cause nullpointexception and other bad consequence
[
https://issues.apache.org/jira/browse/HDFS-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Junjun updated HDFS-4424:
Attachment: patch.txt
> fsdataset Mkdirs failed cause nullpointexception and other bad
> consequence
> ---
>
> Key: HDFS-4424
> URL: https://issues.apache.org/jira/browse/HDFS-4424
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 1.0.1
>Reporter: Li Junjun
> Attachments: patch.txt
>
>
> File: /hadoop-1.0.1/hdfs/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
> from line 205:
> {code}
> if (children == null || children.length == 0) {
> children = new FSDir[maxBlocksPerDir];
> for (int idx = 0; idx < maxBlocksPerDir; idx++) {
> children[idx] = new FSDir(new File(dir,
> DataStorage.BLOCK_SUBDIR_PREFIX+idx));
> }
> }
> {code}
> in FSDir constructer method if faild ( space full,so mkdir fails), but
> the children still in use !
> the the write comes(after I run balancer ) , when choose FSDir
> line 192:
> File file = children[idx].addBlock(b, src, false, resetIdx);
> cause exceptions like this
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSDir.addBlock(FSDataset.java:192)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSDir.addBlock(FSDataset.java:192)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSDir.addBlock(FSDataset.java:158)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.addBlock(FSDataset.java:495)
> {code}
>
> should it like this
> {code}
> if (children == null || children.length == 0) {
> List childrenList = new ArrayList();
>
> for (int idx = 0; idx < maxBlocksPerDir; idx++) {
> try{
>childrenList .add( new FSDir(new File(dir,
> DataStorage.BLOCK_SUBDIR_PREFIX+idx)));
> }catch(Exception e){
> }
> children = childrenList.toArray();
> }
> }
> {code}
>
> bad consequence , in my cluster ,this datanode's num blocks became 0 .
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-347) DFS read performance suboptimal when client co-located on nodes with data
[
https://issues.apache.org/jira/browse/HDFS-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565017#comment-13565017
]
Tsz Wo (Nicholas), SZE commented on HDFS-347:
-
{code}
//DomainSocket.java
+ static {
+if (SystemUtils.IS_OS_WINDOWS) {
+ loadingFailureReason = "UNIX Domain sockets are not available on
Windows.";
+} else if (!NativeCodeLoader.isNativeCodeLoaded()) {
+ loadingFailureReason = "libhadoop cannot be loaded.";
+} else {
+ String problem = "DomainSocket#anchorNative got error: unknown";
+ try {
+anchorNative();
+problem = null;
+ } catch (Throwable t) {
+problem = "DomainSocket#anchorNative got error: " + t.getMessage();
+ }
+ loadingFailureReason = problem;
+}
+ }
{code}
In the code above, when would "DomainSocket#anchorNative got error: unknown" be
used?
BTW, do you have a design doc somewhere?
> DFS read performance suboptimal when client co-located on nodes with data
> -
>
> Key: HDFS-347
> URL: https://issues.apache.org/jira/browse/HDFS-347
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, hdfs-client, performance
>Reporter: George Porter
>Assignee: Colin Patrick McCabe
> Attachments: all.tsv, BlockReaderLocal1.txt, full.patch,
> HADOOP-4801.1.patch, HADOOP-4801.2.patch, HADOOP-4801.3.patch,
> HDFS-347-016_cleaned.patch, HDFS-347.016.patch, HDFS-347.017.clean.patch,
> HDFS-347.017.patch, HDFS-347.018.clean.patch, HDFS-347.018.patch2,
> HDFS-347.019.patch, HDFS-347.020.patch, HDFS-347.021.patch,
> HDFS-347.022.patch, HDFS-347.024.patch, HDFS-347.025.patch,
> HDFS-347.026.patch, HDFS-347.027.patch, HDFS-347.029.patch,
> HDFS-347.030.patch, HDFS-347.033.patch, HDFS-347.035.patch,
> HDFS-347-branch-20-append.txt, hdfs-347-merge.txt, hdfs-347-merge.txt,
> hdfs-347-merge.txt, hdfs-347.png, hdfs-347.txt, local-reads-doc
>
>
> One of the major strategies Hadoop uses to get scalable data processing is to
> move the code to the data. However, putting the DFS client on the same
> physical node as the data blocks it acts on doesn't improve read performance
> as much as expected.
> After looking at Hadoop and O/S traces (via HADOOP-4049), I think the problem
> is due to the HDFS streaming protocol causing many more read I/O operations
> (iops) than necessary. Consider the case of a DFSClient fetching a 64 MB
> disk block from the DataNode process (running in a separate JVM) running on
> the same machine. The DataNode will satisfy the single disk block request by
> sending data back to the HDFS client in 64-KB chunks. In BlockSender.java,
> this is done in the sendChunk() method, relying on Java's transferTo()
> method. Depending on the host O/S and JVM implementation, transferTo() is
> implemented as either a sendfilev() syscall or a pair of mmap() and write().
> In either case, each chunk is read from the disk by issuing a separate I/O
> operation for each chunk. The result is that the single request for a 64-MB
> block ends up hitting the disk as over a thousand smaller requests for 64-KB
> each.
> Since the DFSClient runs in a different JVM and process than the DataNode,
> shuttling data from the disk to the DFSClient also results in context
> switches each time network packets get sent (in this case, the 64-kb chunk
> turns into a large number of 1500 byte packet send operations). Thus we see
> a large number of context switches for each block send operation.
> I'd like to get some feedback on the best way to address this, but I think
> providing a mechanism for a DFSClient to directly open data blocks that
> happen to be on the same machine. It could do this by examining the set of
> LocatedBlocks returned by the NameNode, marking those that should be resident
> on the local host. Since the DataNode and DFSClient (probably) share the
> same hadoop configuration, the DFSClient should be able to find the files
> holding the block data, and it could directly open them and send data back to
> the client. This would avoid the context switches imposed by the network
> layer, and would allow for much larger read buffers than 64KB, which should
> reduce the number of iops imposed by each read block operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4261) TestBalancerWithNodeGroup times out
[ https://issues.apache.org/jira/browse/HDFS-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565015#comment-13565015 ] Junping Du commented on HDFS-4261: -- Sure. Nicholas. I will work on it soon. > TestBalancerWithNodeGroup times out > --- > > Key: HDFS-4261 > URL: https://issues.apache.org/jira/browse/HDFS-4261 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer >Affects Versions: 1.0.4, 1.1.1, 2.0.2-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Junping Du > Fix For: 3.0.0 > > Attachments: HDFS-4261-branch-1.patch, HDFS-4261-branch-1-v2.patch, > HDFS-4261.patch, HDFS-4261-v2.patch, HDFS-4261-v3.patch, HDFS-4261-v4.patch, > HDFS-4261-v5.patch, HDFS-4261-v6.patch, HDFS-4261-v7.patch, > HDFS-4261-v8.patch, jstack-mac-18567, jstack-win-5488, > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.mac, > > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.win, > test-balancer-with-node-group-timeout.txt > > > When I manually ran TestBalancerWithNodeGroup, it always timed out in my > machine. Looking at the Jerkins report [build > #3573|https://builds.apache.org/job/PreCommit-HDFS-Build/3573//testReport/org.apache.hadoop.hdfs.server.balancer/], > TestBalancerWithNodeGroup somehow was skipped so that the problem was not > detected. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4261) TestBalancerWithNodeGroup times out
[ https://issues.apache.org/jira/browse/HDFS-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565007#comment-13565007 ] Tsz Wo (Nicholas), SZE commented on HDFS-4261: -- Hi Junping, TestBalancerWithNodeGroup timed out in [build #3889|https://builds.apache.org/job/PreCommit-HDFS-Build/3889//testReport/org.apache.hadoop.hdfs.server.balancer/TestBalancerWithNodeGroup/testBalancerWithNodeGroup/]. Could you take a look? > TestBalancerWithNodeGroup times out > --- > > Key: HDFS-4261 > URL: https://issues.apache.org/jira/browse/HDFS-4261 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer >Affects Versions: 1.0.4, 1.1.1, 2.0.2-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Junping Du > Fix For: 3.0.0 > > Attachments: HDFS-4261-branch-1.patch, HDFS-4261-branch-1-v2.patch, > HDFS-4261.patch, HDFS-4261-v2.patch, HDFS-4261-v3.patch, HDFS-4261-v4.patch, > HDFS-4261-v5.patch, HDFS-4261-v6.patch, HDFS-4261-v7.patch, > HDFS-4261-v8.patch, jstack-mac-18567, jstack-win-5488, > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.mac, > > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.win, > test-balancer-with-node-group-timeout.txt > > > When I manually ran TestBalancerWithNodeGroup, it always timed out in my > machine. Looking at the Jerkins report [build > #3573|https://builds.apache.org/job/PreCommit-HDFS-Build/3573//testReport/org.apache.hadoop.hdfs.server.balancer/], > TestBalancerWithNodeGroup somehow was skipped so that the problem was not > detected. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4447) Refactor INodeDirectoryWithSnapshot for sharing the code with INodeFileWithSnapshot
[ https://issues.apache.org/jira/browse/HDFS-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4447: - Attachment: h4447_20130128.patch h4447_20130128.patch: adds SnapshotDiffList. > Refactor INodeDirectoryWithSnapshot for sharing the code with > INodeFileWithSnapshot > --- > > Key: HDFS-4447 > URL: https://issues.apache.org/jira/browse/HDFS-4447 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: h4447_20130128.patch > > > This is a code refactoring for HDFS-4446. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Attachment: HDFS-3598.trunk.patch > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.3-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.3-alpha > > Attachments: HDFS-3598.trunk.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Attachment: (was: HDFS-3598.trunk.patch) > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.3-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.3-alpha > > Attachments: HDFS-3598.trunk.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Status: Patch Available (was: Open) > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.3-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.3-alpha > > Attachments: HDFS-3598.trunk.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Status: Open (was: Patch Available) > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.3-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.3-alpha > > Attachments: HDFS-3598.trunk.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3598) WebHDFS: support file concat
[
https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13564988#comment-13564988
]
Hadoop QA commented on HDFS-3598:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12566878/HDFS-3598.trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3896//console
This message is automatically generated.
> WebHDFS: support file concat
>
>
> Key: HDFS-3598
> URL: https://issues.apache.org/jira/browse/HDFS-3598
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.0.3-alpha
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Plamen Jeliazkov
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: HDFS-3598.trunk.patch
>
>
> In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path
> [] psrcs) method. WebHDFS should support it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Status: Patch Available (was: Open) > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.3-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.3-alpha > > Attachments: HDFS-3598.trunk.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Attachment: (was: HDFS-3598.patch) > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.3-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.3-alpha > > Attachments: HDFS-3598.trunk.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Attachment: (was: HDFS-3598.patch) > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.3-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.3-alpha > > Attachments: HDFS-3598.trunk.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Attachment: (was: HDFS-3598.patch) > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.3-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.3-alpha > > Attachments: HDFS-3598.trunk.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Attachment: HDFS-3598.trunk.patch > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.3-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.3-alpha > > Attachments: HDFS-3598.trunk.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Attachment: (was: HDFS-3598.trunk.patch) > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.3-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.3-alpha > > Attachments: HDFS-3598.trunk.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Status: Open (was: Patch Available) > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.3-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.3-alpha > > Attachments: HDFS-3598.trunk.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3598) WebHDFS: support file concat
[
https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13564966#comment-13564966
]
Hadoop QA commented on HDFS-3598:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12566870/HDFS-3598.trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3895//console
This message is automatically generated.
> WebHDFS: support file concat
>
>
> Key: HDFS-3598
> URL: https://issues.apache.org/jira/browse/HDFS-3598
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.0.3-alpha
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Plamen Jeliazkov
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: HDFS-3598.patch, HDFS-3598.patch, HDFS-3598.patch,
> HDFS-3598.trunk.patch
>
>
> In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path
> [] psrcs) method. WebHDFS should support it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4447) Refactor INodeDirectoryWithSnapshot for sharing the code with INodeFileWithSnapshot
Tsz Wo (Nicholas), SZE created HDFS-4447: Summary: Refactor INodeDirectoryWithSnapshot for sharing the code with INodeFileWithSnapshot Key: HDFS-4447 URL: https://issues.apache.org/jira/browse/HDFS-4447 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE This is a code refactoring for HDFS-4446. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Status: Patch Available (was: Open) > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.3-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.3-alpha > > Attachments: HDFS-3598.patch, HDFS-3598.patch, HDFS-3598.patch, > HDFS-3598.trunk.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Attachment: HDFS-3598.trunk.patch > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.3-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.3-alpha > > Attachments: HDFS-3598.patch, HDFS-3598.patch, HDFS-3598.patch, > HDFS-3598.trunk.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4446) Support file snapshot with diff lists
Tsz Wo (Nicholas), SZE created HDFS-4446: Summary: Support file snapshot with diff lists Key: HDFS-4446 URL: https://issues.apache.org/jira/browse/HDFS-4446 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Similar to INodeDirectoryWithSnapshot, it is better to implement INodeFileWithSnapshot so that the INodeFileSnapshot class can be eliminated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Status: Open (was: Patch Available) > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.3-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.3-alpha > > Attachments: HDFS-3598.patch, HDFS-3598.patch, HDFS-3598.patch, > HDFS-3598.trunk.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4131) Add a tool to print the diff between two snapshots and diff of a snapshot from the current tree
[ https://issues.apache.org/jira/browse/HDFS-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4131: Description: This jira provides internal data structures and computation processes for calculating and representing the diff between two snapshots, or the diff between a snapshot and the current tree. (was: This jira provides internal data structures and computation processes for calculating the diff between two snapshots (or the diff between a snapshot and the current tree).) > Add a tool to print the diff between two snapshots and diff of a snapshot > from the current tree > --- > > Key: HDFS-4131 > URL: https://issues.apache.org/jira/browse/HDFS-4131 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: Snapshot (HDFS-2802) >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Attachments: HDFS-4131.001.patch, HDFS-4131.002.patch, > HDFS-4131.003.patch, HDFS-4131.004.patch > > > This jira provides internal data structures and computation processes for > calculating and representing the diff between two snapshots, or the diff > between a snapshot and the current tree. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4131) Add a tool to print the diff between two snapshots and diff of a snapshot from the current tree
[ https://issues.apache.org/jira/browse/HDFS-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4131: Description: This jira provides internal data structures and computation processes for calculating the diff between two snapshots (or the diff between a snapshot and the current tree). (was: This jira tracks tool to print diff between an two snapshots at a given path. The tool will also print the difference between the current directory and the given snapshot.) > Add a tool to print the diff between two snapshots and diff of a snapshot > from the current tree > --- > > Key: HDFS-4131 > URL: https://issues.apache.org/jira/browse/HDFS-4131 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: Snapshot (HDFS-2802) >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Attachments: HDFS-4131.001.patch, HDFS-4131.002.patch, > HDFS-4131.003.patch, HDFS-4131.004.patch > > > This jira provides internal data structures and computation processes for > calculating the diff between two snapshots (or the diff between a snapshot > and the current tree). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4131) Add a tool to print the diff between two snapshots and diff of a snapshot from the current tree
[ https://issues.apache.org/jira/browse/HDFS-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13564890#comment-13564890 ] Suresh Srinivas commented on HDFS-4131: --- Can you please add details of what this patch does? > Add a tool to print the diff between two snapshots and diff of a snapshot > from the current tree > --- > > Key: HDFS-4131 > URL: https://issues.apache.org/jira/browse/HDFS-4131 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: Snapshot (HDFS-2802) >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Attachments: HDFS-4131.001.patch, HDFS-4131.002.patch, > HDFS-4131.003.patch, HDFS-4131.004.patch > > > This jira tracks tool to print diff between an two snapshots at a given path. > The tool will also print the difference between the current directory and the > given snapshot. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4131) Add a tool to print the diff between two snapshots and diff of a snapshot from the current tree
[ https://issues.apache.org/jira/browse/HDFS-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4131: Attachment: HDFS-4131.004.patch Rebase the patch. > Add a tool to print the diff between two snapshots and diff of a snapshot > from the current tree > --- > > Key: HDFS-4131 > URL: https://issues.apache.org/jira/browse/HDFS-4131 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: Snapshot (HDFS-2802) >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Attachments: HDFS-4131.001.patch, HDFS-4131.002.patch, > HDFS-4131.003.patch, HDFS-4131.004.patch > > > This jira tracks tool to print diff between an two snapshots at a given path. > The tool will also print the difference between the current directory and the > given snapshot. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3598) WebHDFS: support file concat
[
https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13564863#comment-13564863
]
Hadoop QA commented on HDFS-3598:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12566856/HDFS-3598.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3894//console
This message is automatically generated.
> WebHDFS: support file concat
>
>
> Key: HDFS-3598
> URL: https://issues.apache.org/jira/browse/HDFS-3598
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.0.3-alpha
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Plamen Jeliazkov
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: HDFS-3598.patch, HDFS-3598.patch, HDFS-3598.patch
>
>
> In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path
> [] psrcs) method. WebHDFS should support it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Attachment: HDFS-3598.patch > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.3-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.3-alpha > > Attachments: HDFS-3598.patch, HDFS-3598.patch, HDFS-3598.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Status: Patch Available (was: Open) > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.3-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.3-alpha > > Attachments: HDFS-3598.patch, HDFS-3598.patch, HDFS-3598.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Status: Open (was: Patch Available) > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.3-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.3-alpha > > Attachments: HDFS-3598.patch, HDFS-3598.patch, HDFS-3598.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3598) WebHDFS: support file concat
[
https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13564849#comment-13564849
]
Hadoop QA commented on HDFS-3598:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12566853/HDFS-3598.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3893//console
This message is automatically generated.
> WebHDFS: support file concat
>
>
> Key: HDFS-3598
> URL: https://issues.apache.org/jira/browse/HDFS-3598
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.0.3-alpha
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Plamen Jeliazkov
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: HDFS-3598.patch, HDFS-3598.patch
>
>
> In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path
> [] psrcs) method. WebHDFS should support it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Status: Patch Available (was: Open) > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.3-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.3-alpha > > Attachments: HDFS-3598.patch, HDFS-3598.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Attachment: HDFS-3598.patch > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.3-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.3-alpha > > Attachments: HDFS-3598.patch, HDFS-3598.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Status: Open (was: Patch Available) > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.3-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.3-alpha > > Attachments: HDFS-3598.patch, HDFS-3598.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4432) Support INodeFileUnderConstructionWithSnapshot in FSImage saving/loading
[ https://issues.apache.org/jira/browse/HDFS-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4432. --- Resolution: Fixed Fix Version/s: Snapshot (HDFS-2802) Hadoop Flags: Reviewed I committed the patch. Thank you Jing! > Support INodeFileUnderConstructionWithSnapshot in FSImage saving/loading > > > Key: HDFS-4432 > URL: https://issues.apache.org/jira/browse/HDFS-4432 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: Snapshot (HDFS-2802) > > Attachments: HDFS-4432.001.patch, HDFS-4432.002.patch, > HDFS-4432.003.patch > > > 1. FSImage saver/loader need to be able to recognize > INodeFileUnderConstruction, INodeFileUnderConstructionWithSnapshot, and > INodeFileUnderConstructionSnapshot. > 2. FSImage saver/loader should be able to form the correct circular linked > list for INodeFileUnderConstruction(With)Snapshot. > 3. Add new corresponding unit tests and support file appending in > TestSnapshot#testSnapshot. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Status: Open (was: Patch Available) > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.3-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.3-alpha > > Attachments: HDFS-3598.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Status: Patch Available (was: Open) > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.3-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.3-alpha > > Attachments: HDFS-3598.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Affects Version/s: 2.0.3-alpha Fix Version/s: 2.0.3-alpha 3.0.0 > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.3-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.3-alpha > > Attachments: HDFS-3598.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3598) WebHDFS: support file concat
[
https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13564818#comment-13564818
]
Hadoop QA commented on HDFS-3598:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12566846/HDFS-3598.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3892//console
This message is automatically generated.
> WebHDFS: support file concat
>
>
> Key: HDFS-3598
> URL: https://issues.apache.org/jira/browse/HDFS-3598
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Plamen Jeliazkov
> Attachments: HDFS-3598.patch
>
>
> In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path
> [] psrcs) method. WebHDFS should support it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Status: Patch Available (was: In Progress) This time created patch with Git. This better work... > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Attachments: HDFS-3598.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-3598 started by Plamen Jeliazkov. > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Attachments: HDFS-3598.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Attachment: (was: hdfs-3598.patch) > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Attachments: HDFS-3598.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Attachment: (was: HDFS-3598.patch) > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Attachments: HDFS-3598.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Attachment: HDFS-3598.patch > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Attachments: HDFS-3598.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Status: Open (was: Patch Available) > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Attachments: HDFS-3598.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3598) WebHDFS: support file concat
[
https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13564786#comment-13564786
]
Hadoop QA commented on HDFS-3598:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12566839/hdfs-3598.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3891//console
This message is automatically generated.
> WebHDFS: support file concat
>
>
> Key: HDFS-3598
> URL: https://issues.apache.org/jira/browse/HDFS-3598
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Plamen Jeliazkov
> Attachments: hdfs-3598.patch, HDFS-3598.patch
>
>
> In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path
> [] psrcs) method. WebHDFS should support it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Attachment: hdfs-3598.patch Retry. Not sure why first patching failing... > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Attachments: hdfs-3598.patch, HDFS-3598.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Status: Patch Available (was: Open) > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Attachments: hdfs-3598.patch, HDFS-3598.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Status: Open (was: Patch Available) > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Attachments: HDFS-3598.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4432) Support INodeFileUnderConstructionWithSnapshot in FSImage saving/loading
[ https://issues.apache.org/jira/browse/HDFS-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13564765#comment-13564765 ] Suresh Srinivas commented on HDFS-4432: --- +1 for the change. > Support INodeFileUnderConstructionWithSnapshot in FSImage saving/loading > > > Key: HDFS-4432 > URL: https://issues.apache.org/jira/browse/HDFS-4432 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-4432.001.patch, HDFS-4432.002.patch, > HDFS-4432.003.patch > > > 1. FSImage saver/loader need to be able to recognize > INodeFileUnderConstruction, INodeFileUnderConstructionWithSnapshot, and > INodeFileUnderConstructionSnapshot. > 2. FSImage saver/loader should be able to form the correct circular linked > list for INodeFileUnderConstruction(With)Snapshot. > 3. Add new corresponding unit tests and support file appending in > TestSnapshot#testSnapshot. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4432) Support INodeFileUnderConstructionWithSnapshot in FSImage saving/loading
[ https://issues.apache.org/jira/browse/HDFS-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4432: Attachment: HDFS-4432.003.patch Thanks for the comments Suresh! The new patch addressed your comments. > Support INodeFileUnderConstructionWithSnapshot in FSImage saving/loading > > > Key: HDFS-4432 > URL: https://issues.apache.org/jira/browse/HDFS-4432 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-4432.001.patch, HDFS-4432.002.patch, > HDFS-4432.003.patch > > > 1. FSImage saver/loader need to be able to recognize > INodeFileUnderConstruction, INodeFileUnderConstructionWithSnapshot, and > INodeFileUnderConstructionSnapshot. > 2. FSImage saver/loader should be able to form the correct circular linked > list for INodeFileUnderConstruction(With)Snapshot. > 3. Add new corresponding unit tests and support file appending in > TestSnapshot#testSnapshot. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3598) WebHDFS: support file concat
[
https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13564751#comment-13564751
]
Hadoop QA commented on HDFS-3598:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12566833/HDFS-3598.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3890//console
This message is automatically generated.
> WebHDFS: support file concat
>
>
> Key: HDFS-3598
> URL: https://issues.apache.org/jira/browse/HDFS-3598
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Plamen Jeliazkov
> Attachments: HDFS-3598.patch
>
>
> In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path
> [] psrcs) method. WebHDFS should support it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Target Version/s: 3.0.0, 2.0.3-alpha Status: Patch Available (was: Open) > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Attachments: HDFS-3598.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3598: --- Attachment: HDFS-3598.patch First patch for review. Tested locally -- appears to be in working order. Added a basic unit test to test functionality. mvn -Dtest=TestFSMainOperationsWebHdfs#testConcat test > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > Attachments: HDFS-3598.patch > > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-347) DFS read performance suboptimal when client co-located on nodes with data
[
https://issues.apache.org/jira/browse/HDFS-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13564737#comment-13564737
]
Colin Patrick McCabe commented on HDFS-347:
---
Looks like all the "new compiler warnings" came from this:
{code}
0a1,8
> [WARNING]
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.hadoop:hadoop-common:jar:3.0.0-SNAPSHOT
> [WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but
> found duplicate declaration of plugin
> org.apache.maven.plugins:maven-surefire-plugin @ line 484, column 15
> [WARNING]
> [WARNING] It is highly recommended to fix these problems because they
> threaten the stability of your build.
> [WARNING]
> [WARNING] For this reason, future Maven versions might no longer support
> building such malformed projects.
> [WARNING]
{code}
It seems that this is a pom.xml problem that was fixed in trunk by HADOOP-9242,
but not in this branch.
As for TestBalancerWithNodeGroup, it's known to be a flaky test-- see HDFS-4376
and HDFS-4261. This branch doesn't change the balancer at all.
> DFS read performance suboptimal when client co-located on nodes with data
> -
>
> Key: HDFS-347
> URL: https://issues.apache.org/jira/browse/HDFS-347
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, hdfs-client, performance
>Reporter: George Porter
>Assignee: Colin Patrick McCabe
> Attachments: all.tsv, BlockReaderLocal1.txt, full.patch,
> HADOOP-4801.1.patch, HADOOP-4801.2.patch, HADOOP-4801.3.patch,
> HDFS-347-016_cleaned.patch, HDFS-347.016.patch, HDFS-347.017.clean.patch,
> HDFS-347.017.patch, HDFS-347.018.clean.patch, HDFS-347.018.patch2,
> HDFS-347.019.patch, HDFS-347.020.patch, HDFS-347.021.patch,
> HDFS-347.022.patch, HDFS-347.024.patch, HDFS-347.025.patch,
> HDFS-347.026.patch, HDFS-347.027.patch, HDFS-347.029.patch,
> HDFS-347.030.patch, HDFS-347.033.patch, HDFS-347.035.patch,
> HDFS-347-branch-20-append.txt, hdfs-347-merge.txt, hdfs-347-merge.txt,
> hdfs-347-merge.txt, hdfs-347.png, hdfs-347.txt, local-reads-doc
>
>
> One of the major strategies Hadoop uses to get scalable data processing is to
> move the code to the data. However, putting the DFS client on the same
> physical node as the data blocks it acts on doesn't improve read performance
> as much as expected.
> After looking at Hadoop and O/S traces (via HADOOP-4049), I think the problem
> is due to the HDFS streaming protocol causing many more read I/O operations
> (iops) than necessary. Consider the case of a DFSClient fetching a 64 MB
> disk block from the DataNode process (running in a separate JVM) running on
> the same machine. The DataNode will satisfy the single disk block request by
> sending data back to the HDFS client in 64-KB chunks. In BlockSender.java,
> this is done in the sendChunk() method, relying on Java's transferTo()
> method. Depending on the host O/S and JVM implementation, transferTo() is
> implemented as either a sendfilev() syscall or a pair of mmap() and write().
> In either case, each chunk is read from the disk by issuing a separate I/O
> operation for each chunk. The result is that the single request for a 64-MB
> block ends up hitting the disk as over a thousand smaller requests for 64-KB
> each.
> Since the DFSClient runs in a different JVM and process than the DataNode,
> shuttling data from the disk to the DFSClient also results in context
> switches each time network packets get sent (in this case, the 64-kb chunk
> turns into a large number of 1500 byte packet send operations). Thus we see
> a large number of context switches for each block send operation.
> I'd like to get some feedback on the best way to address this, but I think
> providing a mechanism for a DFSClient to directly open data blocks that
> happen to be on the same machine. It could do this by examining the set of
> LocatedBlocks returned by the NameNode, marking those that should be resident
> on the local host. Since the DataNode and DFSClient (probably) share the
> same hadoop configuration, the DFSClient should be able to find the files
> holding the block data, and it could directly open them and send data back to
> the client. This would avoid the context switches imposed by the network
> layer, and would allow for much larger read buffers than 64KB, which should
> reduce the number of iops imposed by each read block operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-347) DFS read performance suboptimal when client co-located on nodes with data
[
https://issues.apache.org/jira/browse/HDFS-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13564693#comment-13564693
]
Hadoop QA commented on HDFS-347:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12566804/full.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 22 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 2021 javac
compiler warnings (more than the trunk's current 2013 warnings).
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy:
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/3889//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/3889//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3889//console
This message is automatically generated.
> DFS read performance suboptimal when client co-located on nodes with data
> -
>
> Key: HDFS-347
> URL: https://issues.apache.org/jira/browse/HDFS-347
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, hdfs-client, performance
>Reporter: George Porter
>Assignee: Colin Patrick McCabe
> Attachments: all.tsv, BlockReaderLocal1.txt, full.patch,
> HADOOP-4801.1.patch, HADOOP-4801.2.patch, HADOOP-4801.3.patch,
> HDFS-347-016_cleaned.patch, HDFS-347.016.patch, HDFS-347.017.clean.patch,
> HDFS-347.017.patch, HDFS-347.018.clean.patch, HDFS-347.018.patch2,
> HDFS-347.019.patch, HDFS-347.020.patch, HDFS-347.021.patch,
> HDFS-347.022.patch, HDFS-347.024.patch, HDFS-347.025.patch,
> HDFS-347.026.patch, HDFS-347.027.patch, HDFS-347.029.patch,
> HDFS-347.030.patch, HDFS-347.033.patch, HDFS-347.035.patch,
> HDFS-347-branch-20-append.txt, hdfs-347-merge.txt, hdfs-347-merge.txt,
> hdfs-347-merge.txt, hdfs-347.png, hdfs-347.txt, local-reads-doc
>
>
> One of the major strategies Hadoop uses to get scalable data processing is to
> move the code to the data. However, putting the DFS client on the same
> physical node as the data blocks it acts on doesn't improve read performance
> as much as expected.
> After looking at Hadoop and O/S traces (via HADOOP-4049), I think the problem
> is due to the HDFS streaming protocol causing many more read I/O operations
> (iops) than necessary. Consider the case of a DFSClient fetching a 64 MB
> disk block from the DataNode process (running in a separate JVM) running on
> the same machine. The DataNode will satisfy the single disk block request by
> sending data back to the HDFS client in 64-KB chunks. In BlockSender.java,
> this is done in the sendChunk() method, relying on Java's transferTo()
> method. Depending on the host O/S and JVM implementation, transferTo() is
> implemented as either a sendfilev() syscall or a pair of mmap() and write().
> In either case, each chunk is read from the disk by issuing a separate I/O
> operation for each chunk. The result is that the single request for a 64-MB
> block ends up hitting the disk as over a thousand smaller requests for 64-KB
> each.
> Since the DFSClient runs in a different JVM and process than the DataNode,
> shuttling data from the disk to the DFSClient also results in context
> switches each time network packets get sent (in this case, the 64-kb chunk
> turns into a large number of 1500 byte packet send operations). Thus we see
> a large number of context switches for each block send operation.
> I'd like to get some feedback on the best way to address this, but I think
> providing a mechanism for a DFSClient to directly open data blocks that
> happen to be on the same machine. It could do this by examining the set of
> LocatedBlocks returned by the NameNode, marking those that should
[jira] [Commented] (HDFS-4428) FsDatasetImpl should disclose what the error is when a rename fails
[
https://issues.apache.org/jira/browse/HDFS-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13564678#comment-13564678
]
Chris Nauroth commented on HDFS-4428:
-
+1
Version 2 of the patch addresses everything I mentioned. I applied it locally
and ran {{TestNativeIO}} and {{TestRBWBlockInvalidation}}. Thanks, Colin!
> FsDatasetImpl should disclose what the error is when a rename fails
> ---
>
> Key: HDFS-4428
> URL: https://issues.apache.org/jira/browse/HDFS-4428
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.3-alpha
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Attachments: HDFS-4428.001.patch, HDFS-4428.002.patch
>
>
> It would be nice if {{FsDatasetImpl}} would print out an error message when a
> rename fails, describing what went wrong. This would make it a lot easier to
> investigate and resolve test failures like HDFS-4051.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4428) FsDatasetImpl should disclose what the error is when a rename fails
[
https://issues.apache.org/jira/browse/HDFS-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13564679#comment-13564679
]
Hadoop QA commented on HDFS-4428:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12566803/HDFS-4428.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/3888//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3888//console
This message is automatically generated.
> FsDatasetImpl should disclose what the error is when a rename fails
> ---
>
> Key: HDFS-4428
> URL: https://issues.apache.org/jira/browse/HDFS-4428
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.3-alpha
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Attachments: HDFS-4428.001.patch, HDFS-4428.002.patch
>
>
> It would be nice if {{FsDatasetImpl}} would print out an error message when a
> rename fails, describing what went wrong. This would make it a lot easier to
> investigate and resolve test failures like HDFS-4051.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4131) Add a tool to print the diff between two snapshots and diff of a snapshot from the current tree
[ https://issues.apache.org/jira/browse/HDFS-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13564631#comment-13564631 ] Suresh Srinivas commented on HDFS-4131: --- This patch does not apply on top of latest branch. Needs rebase? > Add a tool to print the diff between two snapshots and diff of a snapshot > from the current tree > --- > > Key: HDFS-4131 > URL: https://issues.apache.org/jira/browse/HDFS-4131 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: Snapshot (HDFS-2802) >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Attachments: HDFS-4131.001.patch, HDFS-4131.002.patch, > HDFS-4131.003.patch > > > This jira tracks tool to print diff between an two snapshots at a given path. > The tool will also print the difference between the current directory and the > given snapshot. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4432) Support INodeFileUnderConstructionWithSnapshot in FSImage saving/loading
[ https://issues.apache.org/jira/browse/HDFS-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13564596#comment-13564596 ] Suresh Srinivas commented on HDFS-4432: --- Can you add an example of the content of the file that is expected in the javadoc of compareDumpedTreeInFile() or link to the method or javadoc where the format is described. > Support INodeFileUnderConstructionWithSnapshot in FSImage saving/loading > > > Key: HDFS-4432 > URL: https://issues.apache.org/jira/browse/HDFS-4432 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-4432.001.patch, HDFS-4432.002.patch > > > 1. FSImage saver/loader need to be able to recognize > INodeFileUnderConstruction, INodeFileUnderConstructionWithSnapshot, and > INodeFileUnderConstructionSnapshot. > 2. FSImage saver/loader should be able to form the correct circular linked > list for INodeFileUnderConstruction(With)Snapshot. > 3. Add new corresponding unit tests and support file appending in > TestSnapshot#testSnapshot. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-347) DFS read performance suboptimal when client co-located on nodes with data
[ https://issues.apache.org/jira/browse/HDFS-347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-347: -- Attachment: full.patch > DFS read performance suboptimal when client co-located on nodes with data > - > > Key: HDFS-347 > URL: https://issues.apache.org/jira/browse/HDFS-347 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, hdfs-client, performance >Reporter: George Porter >Assignee: Colin Patrick McCabe > Attachments: all.tsv, BlockReaderLocal1.txt, full.patch, > HADOOP-4801.1.patch, HADOOP-4801.2.patch, HADOOP-4801.3.patch, > HDFS-347-016_cleaned.patch, HDFS-347.016.patch, HDFS-347.017.clean.patch, > HDFS-347.017.patch, HDFS-347.018.clean.patch, HDFS-347.018.patch2, > HDFS-347.019.patch, HDFS-347.020.patch, HDFS-347.021.patch, > HDFS-347.022.patch, HDFS-347.024.patch, HDFS-347.025.patch, > HDFS-347.026.patch, HDFS-347.027.patch, HDFS-347.029.patch, > HDFS-347.030.patch, HDFS-347.033.patch, HDFS-347.035.patch, > HDFS-347-branch-20-append.txt, hdfs-347-merge.txt, hdfs-347-merge.txt, > hdfs-347-merge.txt, hdfs-347.png, hdfs-347.txt, local-reads-doc > > > One of the major strategies Hadoop uses to get scalable data processing is to > move the code to the data. However, putting the DFS client on the same > physical node as the data blocks it acts on doesn't improve read performance > as much as expected. > After looking at Hadoop and O/S traces (via HADOOP-4049), I think the problem > is due to the HDFS streaming protocol causing many more read I/O operations > (iops) than necessary. Consider the case of a DFSClient fetching a 64 MB > disk block from the DataNode process (running in a separate JVM) running on > the same machine. The DataNode will satisfy the single disk block request by > sending data back to the HDFS client in 64-KB chunks. In BlockSender.java, > this is done in the sendChunk() method, relying on Java's transferTo() > method. Depending on the host O/S and JVM implementation, transferTo() is > implemented as either a sendfilev() syscall or a pair of mmap() and write(). > In either case, each chunk is read from the disk by issuing a separate I/O > operation for each chunk. The result is that the single request for a 64-MB > block ends up hitting the disk as over a thousand smaller requests for 64-KB > each. > Since the DFSClient runs in a different JVM and process than the DataNode, > shuttling data from the disk to the DFSClient also results in context > switches each time network packets get sent (in this case, the 64-kb chunk > turns into a large number of 1500 byte packet send operations). Thus we see > a large number of context switches for each block send operation. > I'd like to get some feedback on the best way to address this, but I think > providing a mechanism for a DFSClient to directly open data blocks that > happen to be on the same machine. It could do this by examining the set of > LocatedBlocks returned by the NameNode, marking those that should be resident > on the local host. Since the DataNode and DFSClient (probably) share the > same hadoop configuration, the DFSClient should be able to find the files > holding the block data, and it could directly open them and send data back to > the client. This would avoid the context switches imposed by the network > layer, and would allow for much larger read buffers than 64KB, which should > reduce the number of iops imposed by each read block operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4444) Add space between total transaction time and number of transactions in FSEditLog#printStatistics
[
https://issues.apache.org/jira/browse/HDFS-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13564552#comment-13564552
]
Hudson commented on HDFS-:
--
Integrated in Hadoop-trunk-Commit #3287 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/3287/])
HDFS-. Add space between total transaction time and number of
transactions in FSEditLog#printStatistics. Contributed by Stephen Chu.
(Revision 1439559)
Result = SUCCESS
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1439559
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
> Add space between total transaction time and number of transactions in
> FSEditLog#printStatistics
>
>
> Key: HDFS-
> URL: https://issues.apache.org/jira/browse/HDFS-
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Stephen Chu
>Assignee: Stephen Chu
>Priority: Trivial
> Fix For: 1.2.0, 2.0.3-alpha
>
> Attachments: HDFS-.patch.001, HDFS-.patch.branch-1
>
>
> Currently, when we log statistics, we see something like
> {code}
> 13/01/25 23:16:59 INFO namenode.FSNamesystem: Number of transactions: 0 Total
> time for transactions(ms): 0Number of transactions batched in Syncs: 0 Number
> of syncs: 0 SyncTimes(ms): 0
> {code}
> Notice how the value for total transactions time and "Number of transactions
> batched in Syncs" needs a space to separate them.
> FSEditLog#printStatistics:
> {code}
> private void printStatistics(boolean force) {
> long now = now();
> if (lastPrintTime + 6 > now && !force) {
> return;
> }
> lastPrintTime = now;
> StringBuilder buf = new StringBuilder();
> buf.append("Number of transactions: ");
> buf.append(numTransactions);
> buf.append(" Total time for transactions(ms): ");
> buf.append(totalTimeTransactions);
> buf.append("Number of transactions batched in Syncs: ");
> buf.append(numTransactionsBatchedInSync);
> buf.append(" Number of syncs: ");
> buf.append(editLogStream.getNumSync());
> buf.append(" SyncTimes(ms): ");
> buf.append(journalSet.getSyncTimes());
> LOG.info(buf);
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4428) FsDatasetImpl should disclose what the error is when a rename fails
[
https://issues.apache.org/jira/browse/HDFS-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-4428:
---
Attachment: HDFS-4428.002.patch
thanks for the thorough review, Chris. This should address those points.
> FsDatasetImpl should disclose what the error is when a rename fails
> ---
>
> Key: HDFS-4428
> URL: https://issues.apache.org/jira/browse/HDFS-4428
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.3-alpha
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Attachments: HDFS-4428.001.patch, HDFS-4428.002.patch
>
>
> It would be nice if {{FsDatasetImpl}} would print out an error message when a
> rename fails, describing what went wrong. This would make it a lot easier to
> investigate and resolve test failures like HDFS-4051.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4108) In a secure cluster, in the HDFS WEBUI , clicking on a datanode in the node list , gives an error
[ https://issues.apache.org/jira/browse/HDFS-4108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-4108: --- Attachment: HDFS-4108-1-2.patch Attaching the patch without dependency on MAPREDUCE-4661 > In a secure cluster, in the HDFS WEBUI , clicking on a datanode in the node > list , gives an error > - > > Key: HDFS-4108 > URL: https://issues.apache.org/jira/browse/HDFS-4108 > Project: Hadoop HDFS > Issue Type: Bug > Components: security, webhdfs >Affects Versions: 1.1.0 >Reporter: Benoy Antony >Assignee: Benoy Antony >Priority: Minor > Attachments: HDFS-4108-1-1.patch, HDFS-4108-1-1.patch, > HDFS-4108-1-2.patch > > > This issue happens in secure cluster. > To reproduce : > Go to the NameNode WEB UI. (dfshealth.jsp) > Click to bring up the list of LiveNodes (dfsnodelist.jsp) > Click on a datanode to bring up the filesystem web page ( > browsedirectory.jsp) > The page containing the directory listing does not come up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4444) Add space between total transaction time and number of transactions in FSEditLog#printStatistics
[
https://issues.apache.org/jira/browse/HDFS-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas updated HDFS-:
--
Resolution: Fixed
Fix Version/s: 2.0.3-alpha
1.2.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed the patch to trunk, branch-1 and branch-2.
Thank you Stephen!
> Add space between total transaction time and number of transactions in
> FSEditLog#printStatistics
>
>
> Key: HDFS-
> URL: https://issues.apache.org/jira/browse/HDFS-
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Stephen Chu
>Assignee: Stephen Chu
>Priority: Trivial
> Fix For: 1.2.0, 2.0.3-alpha
>
> Attachments: HDFS-.patch.001, HDFS-.patch.branch-1
>
>
> Currently, when we log statistics, we see something like
> {code}
> 13/01/25 23:16:59 INFO namenode.FSNamesystem: Number of transactions: 0 Total
> time for transactions(ms): 0Number of transactions batched in Syncs: 0 Number
> of syncs: 0 SyncTimes(ms): 0
> {code}
> Notice how the value for total transactions time and "Number of transactions
> batched in Syncs" needs a space to separate them.
> FSEditLog#printStatistics:
> {code}
> private void printStatistics(boolean force) {
> long now = now();
> if (lastPrintTime + 6 > now && !force) {
> return;
> }
> lastPrintTime = now;
> StringBuilder buf = new StringBuilder();
> buf.append("Number of transactions: ");
> buf.append(numTransactions);
> buf.append(" Total time for transactions(ms): ");
> buf.append(totalTimeTransactions);
> buf.append("Number of transactions batched in Syncs: ");
> buf.append(numTransactionsBatchedInSync);
> buf.append(" Number of syncs: ");
> buf.append(editLogStream.getNumSync());
> buf.append(" SyncTimes(ms): ");
> buf.append(journalSet.getSyncTimes());
> LOG.info(buf);
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3598) WebHDFS: support file concat
[ https://issues.apache.org/jira/browse/HDFS-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13564540#comment-13564540 ] Plamen Jeliazkov commented on HDFS-3598: Ok here's what I've got so far: I will add concat to FileSystem. Where needed I will introduce its dummy body to throw an UnsupportedOperationException for concat. In WebHDFS I will implement concat as a POST operation (similar to append). The initial path variable will be the target destination. I will introduce a ConcatSourcesParam class in order to facilitate the multiple FileSystem path parameters required in concatenation (inline with the other Param classes already existing). > WebHDFS: support file concat > > > Key: HDFS-3598 > URL: https://issues.apache.org/jira/browse/HDFS-3598 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Plamen Jeliazkov > > In trunk and branch-2, DistributedFileSystem has a new concat(Path trg, Path > [] psrcs) method. WebHDFS should support it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4432) Support INodeFileUnderConstructionWithSnapshot in FSImage saving/loading
[ https://issues.apache.org/jira/browse/HDFS-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4432: Attachment: HDFS-4432.002.patch Update the javadoc of FSImageFormat. > Support INodeFileUnderConstructionWithSnapshot in FSImage saving/loading > > > Key: HDFS-4432 > URL: https://issues.apache.org/jira/browse/HDFS-4432 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-4432.001.patch, HDFS-4432.002.patch > > > 1. FSImage saver/loader need to be able to recognize > INodeFileUnderConstruction, INodeFileUnderConstructionWithSnapshot, and > INodeFileUnderConstructionSnapshot. > 2. FSImage saver/loader should be able to form the correct circular linked > list for INodeFileUnderConstruction(With)Snapshot. > 3. Add new corresponding unit tests and support file appending in > TestSnapshot#testSnapshot. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HDFS-4444) Add space between total transaction time and number of transactions in FSEditLog#printStatistics
[
https://issues.apache.org/jira/browse/HDFS-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13564530#comment-13564530
]
Suresh Srinivas edited comment on HDFS- at 1/28/13 6:56 PM:
+1 for the change. I will commit it shortly.
was (Author: sureshms):
+! for the change. I will commit it shortly.
> Add space between total transaction time and number of transactions in
> FSEditLog#printStatistics
>
>
> Key: HDFS-
> URL: https://issues.apache.org/jira/browse/HDFS-
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Stephen Chu
>Assignee: Stephen Chu
>Priority: Trivial
> Attachments: HDFS-.patch.001, HDFS-.patch.branch-1
>
>
> Currently, when we log statistics, we see something like
> {code}
> 13/01/25 23:16:59 INFO namenode.FSNamesystem: Number of transactions: 0 Total
> time for transactions(ms): 0Number of transactions batched in Syncs: 0 Number
> of syncs: 0 SyncTimes(ms): 0
> {code}
> Notice how the value for total transactions time and "Number of transactions
> batched in Syncs" needs a space to separate them.
> FSEditLog#printStatistics:
> {code}
> private void printStatistics(boolean force) {
> long now = now();
> if (lastPrintTime + 6 > now && !force) {
> return;
> }
> lastPrintTime = now;
> StringBuilder buf = new StringBuilder();
> buf.append("Number of transactions: ");
> buf.append(numTransactions);
> buf.append(" Total time for transactions(ms): ");
> buf.append(totalTimeTransactions);
> buf.append("Number of transactions batched in Syncs: ");
> buf.append(numTransactionsBatchedInSync);
> buf.append(" Number of syncs: ");
> buf.append(editLogStream.getNumSync());
> buf.append(" SyncTimes(ms): ");
> buf.append(journalSet.getSyncTimes());
> LOG.info(buf);
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4444) Add space between total transaction time and number of transactions in FSEditLog#printStatistics
[
https://issues.apache.org/jira/browse/HDFS-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13564530#comment-13564530
]
Suresh Srinivas commented on HDFS-:
---
+! for the change. I will commit it shortly.
> Add space between total transaction time and number of transactions in
> FSEditLog#printStatistics
>
>
> Key: HDFS-
> URL: https://issues.apache.org/jira/browse/HDFS-
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Stephen Chu
>Assignee: Stephen Chu
>Priority: Trivial
> Attachments: HDFS-.patch.001, HDFS-.patch.branch-1
>
>
> Currently, when we log statistics, we see something like
> {code}
> 13/01/25 23:16:59 INFO namenode.FSNamesystem: Number of transactions: 0 Total
> time for transactions(ms): 0Number of transactions batched in Syncs: 0 Number
> of syncs: 0 SyncTimes(ms): 0
> {code}
> Notice how the value for total transactions time and "Number of transactions
> batched in Syncs" needs a space to separate them.
> FSEditLog#printStatistics:
> {code}
> private void printStatistics(boolean force) {
> long now = now();
> if (lastPrintTime + 6 > now && !force) {
> return;
> }
> lastPrintTime = now;
> StringBuilder buf = new StringBuilder();
> buf.append("Number of transactions: ");
> buf.append(numTransactions);
> buf.append(" Total time for transactions(ms): ");
> buf.append(totalTimeTransactions);
> buf.append("Number of transactions batched in Syncs: ");
> buf.append(numTransactionsBatchedInSync);
> buf.append(" Number of syncs: ");
> buf.append(editLogStream.getNumSync());
> buf.append(" SyncTimes(ms): ");
> buf.append(journalSet.getSyncTimes());
> LOG.info(buf);
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3874) Exception when client reports bad checksum to NN
[
https://issues.apache.org/jira/browse/HDFS-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee resolved HDFS-3874.
--
Resolution: Fixed
> Exception when client reports bad checksum to NN
>
>
> Key: HDFS-3874
> URL: https://issues.apache.org/jira/browse/HDFS-3874
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, namenode
>Affects Versions: 2.0.0-alpha, 0.23.5
>Reporter: Todd Lipcon
>Assignee: Kihwal Lee
>Priority: Critical
>
> We see the following exception in our logs on a cluster:
> {code}
> 2012-08-27 16:34:30,400 INFO org.apache.hadoop.hdfs.StateChange: *DIR*
> NameNode.reportBadBlocks
> 2012-08-27 16:34:30,400 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:hdfs (auth:SIMPLE) cause:java.io.IOException: Cannot mark
> blk_8285012733733669474_140475196{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[172.29.97.219:50010|RBW]]}(same as stored)
> as corrupt because datanode :0 does not exist
> 2012-08-27 16:34:30,400 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 46 on 8020, call
> org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol.reportBadBlocks from
> 172.29.97.219:43805: error: java.io.IOException: Cannot mark
> blk_8285012733733669474_140475196{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[172.29.97.219:50010|RBW]]}(same as stored)
> as corrupt because datanode :0 does not exist
> java.io.IOException: Cannot mark
> blk_8285012733733669474_140475196{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[172.29.97.219:50010|RBW]]}(same as stored)
> as corrupt because datanode :0 does not exist
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.markBlockAsCorrupt(BlockManager.java:1001)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.findAndMarkBlockAsCorrupt(BlockManager.java:994)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.reportBadBlocks(FSNamesystem.java:4736)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.reportBadBlocks(NameNodeRpcServer.java:537)
> at
> org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.reportBadBlocks(DatanodeProtocolServerSideTranslatorPB.java:242)
> at
> org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:20032)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3874) Exception when client reports bad checksum to NN
[
https://issues.apache.org/jira/browse/HDFS-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13564500#comment-13564500
]
Kihwal Lee commented on HDFS-3874:
--
This defect will be fixed as part of HDFS-3875.
> Exception when client reports bad checksum to NN
>
>
> Key: HDFS-3874
> URL: https://issues.apache.org/jira/browse/HDFS-3874
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, namenode
>Affects Versions: 2.0.0-alpha, 0.23.5
>Reporter: Todd Lipcon
>Assignee: Kihwal Lee
>Priority: Critical
>
> We see the following exception in our logs on a cluster:
> {code}
> 2012-08-27 16:34:30,400 INFO org.apache.hadoop.hdfs.StateChange: *DIR*
> NameNode.reportBadBlocks
> 2012-08-27 16:34:30,400 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:hdfs (auth:SIMPLE) cause:java.io.IOException: Cannot mark
> blk_8285012733733669474_140475196{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[172.29.97.219:50010|RBW]]}(same as stored)
> as corrupt because datanode :0 does not exist
> 2012-08-27 16:34:30,400 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 46 on 8020, call
> org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol.reportBadBlocks from
> 172.29.97.219:43805: error: java.io.IOException: Cannot mark
> blk_8285012733733669474_140475196{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[172.29.97.219:50010|RBW]]}(same as stored)
> as corrupt because datanode :0 does not exist
> java.io.IOException: Cannot mark
> blk_8285012733733669474_140475196{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[172.29.97.219:50010|RBW]]}(same as stored)
> as corrupt because datanode :0 does not exist
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.markBlockAsCorrupt(BlockManager.java:1001)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.findAndMarkBlockAsCorrupt(BlockManager.java:994)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.reportBadBlocks(FSNamesystem.java:4736)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.reportBadBlocks(NameNodeRpcServer.java:537)
> at
> org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.reportBadBlocks(DatanodeProtocolServerSideTranslatorPB.java:242)
> at
> org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:20032)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4426) Secondary namenode shuts down immediately after startup
[
https://issues.apache.org/jira/browse/HDFS-4426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves updated HDFS-4426:
Fix Version/s: 0.23.7
> Secondary namenode shuts down immediately after startup
> ---
>
> Key: HDFS-4426
> URL: https://issues.apache.org/jira/browse/HDFS-4426
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.0.3-alpha, 0.23.6
>Reporter: Jason Lowe
>Assignee: Arpit Agarwal
>Priority: Blocker
> Fix For: 2.0.3-alpha, 0.23.6, 0.23.7
>
> Attachments: HDFS-4426.1.patch, HDFS-4426.branch-23.patch,
> HDFS-4426.patch, HDFS-4426.patch, HDFS-4426.patch
>
>
> After HADOOP-9181 went in, the secondary namenode immediately shuts down
> after it is started. From the startup logs:
> {noformat}
> 2013-01-22 19:54:28,826 INFO namenode.SecondaryNameNode
> (SecondaryNameNode.java:initialize(299)) - Checkpoint Period :3600 secs (60
> min)
> 2013-01-22 19:54:28,826 INFO namenode.SecondaryNameNode
> (SecondaryNameNode.java:initialize(301)) - Log Size Trigger:4 txns
> 2013-01-22 19:54:28,845 INFO namenode.SecondaryNameNode
> (StringUtils.java:run(616)) - SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down SecondaryNameNode at xx
> /
> {noformat}
> I looked into the issue, and it's shutting down because
> SecondaryNameNode.main starts a bunch of daemon threads then returns. With
> nothing but daemon threads remaining, the JVM sees no reason to keep going
> and proceeds to shutdown. Apparently we were implicitly relying on the fact
> that the HttpServer QueuedThreadPool threads were not daemon threads to keep
> the secondary namenode process up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4437) Clients should not retain leases on moved files
[ https://issues.apache.org/jira/browse/HDFS-4437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13564346#comment-13564346 ] Daryn Sharp commented on HDFS-4437: --- I think supporting file descriptor behavior is a great idea (we've internally talked about this). Until we do, I think the lease should be revoked. My concerns with fd behavior would be the ever pervasive "two wrongs make a right" where users are relying unintentionally on renames breaking writers, and ensuring we get the security right to avoid attacks probing for fileids. > Clients should not retain leases on moved files > --- > > Key: HDFS-4437 > URL: https://issues.apache.org/jira/browse/HDFS-4437 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > > Moving open files and/or parent directories of open files will rewrite the > leases to the new path. The client is not notified so future stream > operations will fail. However, as long as the client keeps its lease active > it will have a "lock" on the file in its new location. This is not good for > a daemon. > Leases should be released after the file is moved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4259) Improve pipeline DN replacement failure message
[ https://issues.apache.org/jira/browse/HDFS-4259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13564268#comment-13564268 ] Hudson commented on HDFS-4259: -- Integrated in Hadoop-Mapreduce-trunk #1327 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1327/]) HDFS-4259. Improve pipeline DN replacement failure message. Contributed by Harsh J. (harsh) (Revision 1439126) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1439126 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java > Improve pipeline DN replacement failure message > --- > > Key: HDFS-4259 > URL: https://issues.apache.org/jira/browse/HDFS-4259 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 2.0.2-alpha >Reporter: Harsh J >Assignee: Harsh J >Priority: Minor > Fix For: 2.0.3-alpha > > Attachments: HDFS-4259.patch > > > The current message shown is something such as below: > bq. Failed to add a datanode. User may turn off this feature by setting > X.policy in configuration, where the current policy is Y. (Nodes: > current=\[foo\], original=\[bar\]) > This reads off like failing is a feature (but the intention and the reason we > hit this isn't indicated strongly), and can be bettered. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4259) Improve pipeline DN replacement failure message
[ https://issues.apache.org/jira/browse/HDFS-4259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13564250#comment-13564250 ] Hudson commented on HDFS-4259: -- Integrated in Hadoop-Hdfs-trunk #1299 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1299/]) HDFS-4259. Improve pipeline DN replacement failure message. Contributed by Harsh J. (harsh) (Revision 1439126) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1439126 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java > Improve pipeline DN replacement failure message > --- > > Key: HDFS-4259 > URL: https://issues.apache.org/jira/browse/HDFS-4259 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 2.0.2-alpha >Reporter: Harsh J >Assignee: Harsh J >Priority: Minor > Fix For: 2.0.3-alpha > > Attachments: HDFS-4259.patch > > > The current message shown is something such as below: > bq. Failed to add a datanode. User may turn off this feature by setting > X.policy in configuration, where the current policy is Y. (Nodes: > current=\[foo\], original=\[bar\]) > This reads off like failing is a feature (but the intention and the reason we > hit this isn't indicated strongly), and can be bettered. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4259) Improve pipeline DN replacement failure message
[ https://issues.apache.org/jira/browse/HDFS-4259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13564184#comment-13564184 ] Hudson commented on HDFS-4259: -- Integrated in Hadoop-Yarn-trunk #110 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/110/]) HDFS-4259. Improve pipeline DN replacement failure message. Contributed by Harsh J. (harsh) (Revision 1439126) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1439126 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java > Improve pipeline DN replacement failure message > --- > > Key: HDFS-4259 > URL: https://issues.apache.org/jira/browse/HDFS-4259 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 2.0.2-alpha >Reporter: Harsh J >Assignee: Harsh J >Priority: Minor > Fix For: 2.0.3-alpha > > Attachments: HDFS-4259.patch > > > The current message shown is something such as below: > bq. Failed to add a datanode. User may turn off this feature by setting > X.policy in configuration, where the current policy is Y. (Nodes: > current=\[foo\], original=\[bar\]) > This reads off like failing is a feature (but the intention and the reason we > hit this isn't indicated strongly), and can be bettered. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4445) All BKJM ledgers are not checked while tailing, So failover will fail.
[
https://issues.apache.org/jira/browse/HDFS-4445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-4445:
--
Target Version/s: 2.0.3-alpha
> All BKJM ledgers are not checked while tailing, So failover will fail.
> --
>
> Key: HDFS-4445
> URL: https://issues.apache.org/jira/browse/HDFS-4445
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.0.3-alpha
>Reporter: Vinay
>Assignee: Vinay
>Priority: Blocker
>
> After the fix of HDFS-4130, all editlog nodes are not iterated if first edit
> are below fromTxId
> Problem part is below code inside
> BookKeeperJournalManager#selectInputStreams(..)
> {code}if (fromTxId >= l.getFirstTxId() && fromTxId <= lastTxId) {
> LedgerHandle h;
> if (l.isInProgress()) { // we don't want to fence the current
> journal
> h = bkc.openLedgerNoRecovery(l.getLedgerId(), digestType,
> digestpw.getBytes());
> } else {
> h = bkc
> .openLedger(l.getLedgerId(), digestType, digestpw.getBytes());
> }
> elis = new BookKeeperEditLogInputStream(h, l);
> elis.skipTo(fromTxId);
> } else {
> return;
> }{code}
> The else block should have continue statement instead of return.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4432) Support INodeFileUnderConstructionWithSnapshot in FSImage saving/loading
[ https://issues.apache.org/jira/browse/HDFS-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4432: Attachment: HDFS-4432.001.patch Update the patch. > Support INodeFileUnderConstructionWithSnapshot in FSImage saving/loading > > > Key: HDFS-4432 > URL: https://issues.apache.org/jira/browse/HDFS-4432 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-4432.001.patch > > > 1. FSImage saver/loader need to be able to recognize > INodeFileUnderConstruction, INodeFileUnderConstructionWithSnapshot, and > INodeFileUnderConstructionSnapshot. > 2. FSImage saver/loader should be able to form the correct circular linked > list for INodeFileUnderConstruction(With)Snapshot. > 3. Add new corresponding unit tests and support file appending in > TestSnapshot#testSnapshot. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
