[jira] Commented: (HDFS-850) Display more memory details on the web ui
[ https://issues.apache.org/jira/browse/HDFS-850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843947#action_12843947 ] Hudson commented on HDFS-850: - Integrated in Hadoop-Hdfs-trunk-Commit #213 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/213/]) . The WebUI displays more details about namenode memory usage. (Dmytro Molkov via dhruba) > Display more memory details on the web ui > - > > Key: HDFS-850 > URL: https://issues.apache.org/jira/browse/HDFS-850 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.22.0 >Reporter: Dmytro Molkov >Assignee: Dmytro Molkov >Priority: Minor > Fix For: 0.22.0 > > Attachments: HDFS-850.patch, HDFS-850.patch, HDFS-850.patch, > screenshot-1.jpg, screenshot-2.jpg > > > With the HDFS-94 being commited, the namenode will use JMX memory beans to > get information about heap usage. > They provide us with additional information such as NonHeap memory usage and > Heap Commited and Initialized memory in addition to Used and Max. > It will be useful to see that additional information on the NameNode web ui. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-854) Datanode should scan devices in parallel to generate block report
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843935#action_12843935 ] dhruba borthakur commented on HDFS-854: --- > Or probably keep last added/deleted blocks in memory and send block report > and doing disk scan once in a while? Lohit:This is already part of HDFS trunk, I think Suresh has earlier improved this to generate block reports from memory. This patch improves the startup time of the cluster when the entire cluster is restarted. In this case, the datanodes are started newly and have to scan the disk to generate the first block report. The faster we can generate the block report, the shorter the time for the NN to exit safemode! > Datanode should scan devices in parallel to generate block report > - > > Key: HDFS-854 > URL: https://issues.apache.org/jira/browse/HDFS-854 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Reporter: dhruba borthakur >Assignee: Dmytro Molkov > Attachments: HDFS-854.patch > > > A Datanode should scan its disk devices in parallel so that the time to > generate a block report is reduced. This will reduce the startup time of a > cluster. > A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total > of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to > scan these devices to generate the first block report. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-850) Display more memory details on the web ui
[ https://issues.apache.org/jira/browse/HDFS-850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-850: -- Resolution: Fixed Fix Version/s: 0.22.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I just committed this. Thanks Dmytro! > Display more memory details on the web ui > - > > Key: HDFS-850 > URL: https://issues.apache.org/jira/browse/HDFS-850 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.22.0 >Reporter: Dmytro Molkov >Assignee: Dmytro Molkov >Priority: Minor > Fix For: 0.22.0 > > Attachments: HDFS-850.patch, HDFS-850.patch, HDFS-850.patch, > screenshot-1.jpg, screenshot-2.jpg > > > With the HDFS-94 being commited, the namenode will use JMX memory beans to > get information about heap usage. > They provide us with additional information such as NonHeap memory usage and > Heap Commited and Initialized memory in addition to Used and Max. > It will be useful to see that additional information on the NameNode web ui. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-850) Display more memory details on the web ui
[ https://issues.apache.org/jira/browse/HDFS-850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843927#action_12843927 ] dhruba borthakur commented on HDFS-850: --- +1 Code looks good. I will commit it. > Display more memory details on the web ui > - > > Key: HDFS-850 > URL: https://issues.apache.org/jira/browse/HDFS-850 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.22.0 >Reporter: Dmytro Molkov >Assignee: Dmytro Molkov >Priority: Minor > Attachments: HDFS-850.patch, HDFS-850.patch, HDFS-850.patch, > screenshot-1.jpg, screenshot-2.jpg > > > With the HDFS-94 being commited, the namenode will use JMX memory beans to > get information about heap usage. > They provide us with additional information such as NonHeap memory usage and > Heap Commited and Initialized memory in addition to Used and Max. > It will be useful to see that additional information on the NameNode web ui. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-854) Datanode should scan devices in parallel to generate block report
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmytro Molkov updated HDFS-854: --- Attachment: HDFS-854.patch Please have a look at the patch. The problem we are trying to solve here is generating the first block report quicker after restart by scanning the volumes in parallel. This way instead of scanning 12 TB of data sequentially we scan 12 chunks of 1 TB in parallel. Since there is a lot of latency in IO we have an improvement of a few times in the time to generate the block report. The test for this is just running the directory scanner test twice: with parallel execution and without it. > Datanode should scan devices in parallel to generate block report > - > > Key: HDFS-854 > URL: https://issues.apache.org/jira/browse/HDFS-854 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Reporter: dhruba borthakur >Assignee: Dmytro Molkov > Attachments: HDFS-854.patch > > > A Datanode should scan its disk devices in parallel so that the time to > generate a block report is reduced. This will reduce the startup time of a > cluster. > A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total > of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to > scan these devices to generate the first block report. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1033) In secure clusters, NN and SNN should verify that the remote principal during image and edits transfer
[ https://issues.apache.org/jira/browse/HDFS-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1033: -- Attachment: HDFS-1033-Y20.patch Y! 20 patch, not for commit. Trunk patch soon... > In secure clusters, NN and SNN should verify that the remote principal during > image and edits transfer > -- > > Key: HDFS-1033 > URL: https://issues.apache.org/jira/browse/HDFS-1033 > Project: Hadoop HDFS > Issue Type: Improvement > Components: security >Reporter: Jakob Homan >Assignee: Jakob Homan > Attachments: HDFS-1033-Y20.patch > > > Currently anyone can connect and download image/edits from Namenode. In a > secure cluster we can verify the identity of the principal making the > request; we should disallow requests from anyone except the NN and SNN > principals (and their hosts due to the lousy KerbSSL limitation). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1005) Fsck security
[ https://issues.apache.org/jira/browse/HDFS-1005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-1005: -- Attachment: HDFS-1005-BP20-1.patch Bugfix on top of the earlier Y20 patch. > Fsck security > - > > Key: HDFS-1005 > URL: https://issues.apache.org/jira/browse/HDFS-1005 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jitendra Nath Pandey >Assignee: Boris Shkolnik > Attachments: HDFS-1005-BP20-1.patch, HDFS-1005-BP20.patch, > HDFS-1005-y20.1.patch > > > This jira tracks implementation of security for Fsck. Fsck should make an > authenticated connection to the namenode. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1024) SecondaryNamenode fails to checkpoint because namenode fails with CancelledKeyException
[ https://issues.apache.org/jira/browse/HDFS-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843891#action_12843891 ] Dmytro Molkov commented on HDFS-1024: - Note though, that this patch doesn't fix the underlying issue (that might as well be a jetty but of sending huge images) and only protects HDFS from corrupting the image in the case when the partial image read will be valid and will substitute the current image after secondary name node processes it. > SecondaryNamenode fails to checkpoint because namenode fails with > CancelledKeyException > --- > > Key: HDFS-1024 > URL: https://issues.apache.org/jira/browse/HDFS-1024 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.20.2 >Reporter: dhruba borthakur >Assignee: Dmytro Molkov > Attachments: HDFS-1024.patch > > > The secondary namenode fails to retrieve the entire fsimage from the > Namenode. It fetches a part of the fsimage but believes that it has fetched > the entire fsimage file and proceeds ahead with the checkpointing. Stack > traces will be attached below. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1024) SecondaryNamenode fails to checkpoint because namenode fails with CancelledKeyException
[ https://issues.apache.org/jira/browse/HDFS-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmytro Molkov updated HDFS-1024: Attachment: HDFS-1024.patch Since the image file can bee too long sometimes Content-Length is not set, instead it sets Transfer-Encoding: chunked I am setting Content-Length manually in the servlet now. One thing to note here, there is a response.setContentLength(int) method, but taking int as a parameter it will not work, since many images are over 2Gb in size. So I am setting it as a general HTTP header. On the client side once the read is over we check if the number of bytes we read is equal to advertised in the Content-Length and throw if it is not. > SecondaryNamenode fails to checkpoint because namenode fails with > CancelledKeyException > --- > > Key: HDFS-1024 > URL: https://issues.apache.org/jira/browse/HDFS-1024 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.20.2 >Reporter: dhruba borthakur >Assignee: Dmytro Molkov > Attachments: HDFS-1024.patch > > > The secondary namenode fails to retrieve the entire fsimage from the > Namenode. It fetches a part of the fsimage but believes that it has fetched > the entire fsimage file and proceeds ahead with the checkpointing. Stack > traces will be attached below. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1033) In secure clusters, NN and SNN should verify that the remote principal during image and edits transfer
[ https://issues.apache.org/jira/browse/HDFS-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1033: -- Summary: In secure clusters, NN and SNN should verify that the remote principal during image and edits transfer (was: In securre clusters, NN and SNN should verify that the remote principal during image and edits transfer) > In secure clusters, NN and SNN should verify that the remote principal during > image and edits transfer > -- > > Key: HDFS-1033 > URL: https://issues.apache.org/jira/browse/HDFS-1033 > Project: Hadoop HDFS > Issue Type: Improvement > Components: security >Reporter: Jakob Homan >Assignee: Jakob Homan > > Currently anyone can connect and download image/edits from Namenode. In a > secure cluster we can verify the identity of the principal making the > request; we should disallow requests from anyone except the NN and SNN > principals (and their hosts due to the lousy KerbSSL limitation). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-1033) In securre clusters, NN and SNN should verify that the remote principal during image and edits transfer
In securre clusters, NN and SNN should verify that the remote principal during image and edits transfer --- Key: HDFS-1033 URL: https://issues.apache.org/jira/browse/HDFS-1033 Project: Hadoop HDFS Issue Type: Improvement Components: security Reporter: Jakob Homan Assignee: Jakob Homan Currently anyone can connect and download image/edits from Namenode. In a secure cluster we can verify the identity of the principal making the request; we should disallow requests from anyone except the NN and SNN principals (and their hosts due to the lousy KerbSSL limitation). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-861) fuse-dfs does not support O_RDWR
[ https://issues.apache.org/jira/browse/HDFS-861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843847#action_12843847 ] Hudson commented on HDFS-861: - Integrated in Hadoop-Hdfs-trunk-Commit #212 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/212/]) . fuse-dfs does not support O_RDWR. Contributed by Brian Bockelman. > fuse-dfs does not support O_RDWR > > > Key: HDFS-861 > URL: https://issues.apache.org/jira/browse/HDFS-861 > Project: Hadoop HDFS > Issue Type: Bug > Components: contrib/fuse-dfs >Reporter: Brian Bockelman >Assignee: Brian Bockelman >Priority: Minor > Fix For: 0.22.0 > > Attachments: HDFS-861.patch > > > Some applications (for us, the big one is rsync) will open a file in > read-write mode when it really only intends to read xor write (not both). > fuse-dfs should try to not fail until the application actually tries to write > to a pre-existing file or read from a newly created file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-961) dfs_readdir incorrectly parses paths
[
https://issues.apache.org/jira/browse/HDFS-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843846#action_12843846
]
Hudson commented on HDFS-961:
-
Integrated in Hadoop-Hdfs-trunk-Commit #212 (See
[http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/212/])
. dfs_readdir incorrectly parses paths. Contributed by Eli Collins.
> dfs_readdir incorrectly parses paths
>
>
> Key: HDFS-961
> URL: https://issues.apache.org/jira/browse/HDFS-961
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: contrib/fuse-dfs
>Affects Versions: 0.20.1, 0.20.2, 0.21.0
>Reporter: Eli Collins
>Assignee: Eli Collins
> Fix For: 0.22.0
>
> Attachments: hdfs-961-1.patch, hdfs-961-2.patch
>
>
> fuse-dfs dfs_readdir assumes that DistributedFileSystem#listStatus returns
> Paths with the same scheme/authority as the dfs.name.dir used to connect. If
> NameNode.DEFAULT_PORT port is used listStatus returns Paths that have
> authorities without the port (see HDFS-960), which breaks the following code.
> {code}
> // hack city: todo fix the below to something nicer and more maintainable but
> // with good performance
> // strip off the path but be careful if the path is solely '/'
> // NOTE - this API started returning filenames as full dfs uris
> const char *const str = info[i].mName + dfs->dfs_uri_len + path_len +
> ((path_len == 1 && *path == '/') ? 0 : 1);
> {code}
> Let's make the path parsing here more robust. listStatus returns normalized
> paths so we can find the start of the path by searching for the 3rd slash. A
> more long term solution is to have hdfsFileInfo maintain a path object or at
> least pointers to the relevant URI components.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-858) Incorrect return codes for fuse-dfs
[ https://issues.apache.org/jira/browse/HDFS-858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843836#action_12843836 ] Hudson commented on HDFS-858: - Integrated in Hadoop-Hdfs-trunk-Commit #211 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/211/]) . Incorrect return codes for fuse-dfs. Contributed by Brian Bockelman. > Incorrect return codes for fuse-dfs > --- > > Key: HDFS-858 > URL: https://issues.apache.org/jira/browse/HDFS-858 > Project: Hadoop HDFS > Issue Type: Bug > Components: contrib/fuse-dfs >Reporter: Brian Bockelman >Assignee: Brian Bockelman >Priority: Minor > Fix For: 0.22.0 > > Attachments: HDFS-858-2.patch, HDFS-858.patch > > > fuse-dfs doesn't pass proper error codes from libhdfs; places I'd like to > correct are hdfsFileOpen (which can result in permission denied or quota > violations) and hdfsWrite (which can result in quota violations). > By returning the correct error codes, command line utilities return much > better error messages - especially for quota violations, which can be a devil > to debug. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-859) fuse-dfs utime behavior causes issues with tar
[ https://issues.apache.org/jira/browse/HDFS-859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843837#action_12843837 ] Hudson commented on HDFS-859: - Integrated in Hadoop-Hdfs-trunk-Commit #211 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/211/]) . fuse-dfs utime behavior causes issues with tar. Contributed by Brian Bockelman. > fuse-dfs utime behavior causes issues with tar > -- > > Key: HDFS-859 > URL: https://issues.apache.org/jira/browse/HDFS-859 > Project: Hadoop HDFS > Issue Type: Bug > Components: contrib/fuse-dfs >Reporter: Brian Bockelman >Assignee: Brian Bockelman >Priority: Minor > Fix For: 0.22.0 > > Attachments: HDFS-859-2.patch, HDFS-859.patch > > > When trying to untar files onto fuse-dfs, tar will try to set the utime on > all the files and directories. However, setting the utime on a directory in > libhdfs causes an error. > We should silently ignore the failure of setting a utime on a directory; this > will allow tar to complete successfully. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-961) dfs_readdir incorrectly parses paths
[
https://issues.apache.org/jira/browse/HDFS-961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom White updated HDFS-961:
---
Resolution: Fixed
Fix Version/s: 0.22.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I've just committed this. Thanks Eli!
(Failing tests are unrelated.)
> dfs_readdir incorrectly parses paths
>
>
> Key: HDFS-961
> URL: https://issues.apache.org/jira/browse/HDFS-961
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: contrib/fuse-dfs
>Affects Versions: 0.20.1, 0.20.2, 0.21.0
>Reporter: Eli Collins
>Assignee: Eli Collins
> Fix For: 0.22.0
>
> Attachments: hdfs-961-1.patch, hdfs-961-2.patch
>
>
> fuse-dfs dfs_readdir assumes that DistributedFileSystem#listStatus returns
> Paths with the same scheme/authority as the dfs.name.dir used to connect. If
> NameNode.DEFAULT_PORT port is used listStatus returns Paths that have
> authorities without the port (see HDFS-960), which breaks the following code.
> {code}
> // hack city: todo fix the below to something nicer and more maintainable but
> // with good performance
> // strip off the path but be careful if the path is solely '/'
> // NOTE - this API started returning filenames as full dfs uris
> const char *const str = info[i].mName + dfs->dfs_uri_len + path_len +
> ((path_len == 1 && *path == '/') ? 0 : 1);
> {code}
> Let's make the path parsing here more robust. listStatus returns normalized
> paths so we can find the start of the path by searching for the 3rd slash. A
> more long term solution is to have hdfsFileInfo maintain a path object or at
> least pointers to the relevant URI components.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-861) fuse-dfs does not support O_RDWR
[ https://issues.apache.org/jira/browse/HDFS-861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HDFS-861: --- Resolution: Fixed Fix Version/s: 0.22.0 Assignee: Brian Bockelman Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I've just committed this. Thanks Brian! (The test failure is unrelated.) > fuse-dfs does not support O_RDWR > > > Key: HDFS-861 > URL: https://issues.apache.org/jira/browse/HDFS-861 > Project: Hadoop HDFS > Issue Type: Bug > Components: contrib/fuse-dfs >Reporter: Brian Bockelman >Assignee: Brian Bockelman >Priority: Minor > Fix For: 0.22.0 > > Attachments: HDFS-861.patch > > > Some applications (for us, the big one is rsync) will open a file in > read-write mode when it really only intends to read xor write (not both). > fuse-dfs should try to not fail until the application actually tries to write > to a pre-existing file or read from a newly created file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-856) Hardcoded replication level for new files in fuse-dfs
[ https://issues.apache.org/jira/browse/HDFS-856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843817#action_12843817 ] Hudson commented on HDFS-856: - Integrated in Hadoop-Hdfs-trunk-Commit #210 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/210/]) . Hardcoded replication level for new files in fuse-dfs. Contributed by Brian Bockelman. > Hardcoded replication level for new files in fuse-dfs > - > > Key: HDFS-856 > URL: https://issues.apache.org/jira/browse/HDFS-856 > Project: Hadoop HDFS > Issue Type: Bug > Components: contrib/fuse-dfs >Reporter: Brian Bockelman >Assignee: Brian Bockelman >Priority: Minor > Fix For: 0.22.0 > > Attachments: HADOOP-856.patch > > > In fuse-dfs, the number of replicas is always hardcoded to 3 in the arguments > to hdfsOpenFile. We should use the setting in the hadoop configuration > instead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-857) Incorrect type for fuse-dfs capacity can cause "df" to return negative values on 32-bit machines
[ https://issues.apache.org/jira/browse/HDFS-857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843816#action_12843816 ] Hudson commented on HDFS-857: - Integrated in Hadoop-Hdfs-trunk-Commit #210 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/210/]) . Incorrect type for fuse-dfs capacity can cause "df" to return negative values on 32-bit machines. Contributed by Brian Bockelman. > Incorrect type for fuse-dfs capacity can cause "df" to return negative values > on 32-bit machines > > > Key: HDFS-857 > URL: https://issues.apache.org/jira/browse/HDFS-857 > Project: Hadoop HDFS > Issue Type: Bug > Components: contrib/fuse-dfs >Reporter: Brian Bockelman >Assignee: Brian Bockelman >Priority: Minor > Fix For: 0.22.0 > > Attachments: HDFS-857.patch > > > On sufficiently large HDFS installs, the casting of hdfsGetCapacity to a long > may cause "df" to return negative values. tOffset should be used instead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-860) fuse-dfs truncate behavior causes issues with scp
[ https://issues.apache.org/jira/browse/HDFS-860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843808#action_12843808 ] Tom White commented on HDFS-860: Brian, Another solution would be to return 0 if the size to truncate to is the same as the file's size. This would cover the scp case, without breaking error codes, no? BTW what OS are you using? I haven't been able to reproduce this on Centos or Ubuntu, as scp doesn't seem to be calling truncate. > fuse-dfs truncate behavior causes issues with scp > - > > Key: HDFS-860 > URL: https://issues.apache.org/jira/browse/HDFS-860 > Project: Hadoop HDFS > Issue Type: Bug > Components: contrib/fuse-dfs >Reporter: Brian Bockelman >Priority: Minor > Attachments: HDFS-860.patch > > > For whatever reason, scp issues a "truncate" once it's written a file to > truncate the file to the # of bytes it has written (i.e., if a file is X > bytes, it calls truncate(X)). > This fails on the current fuse-dfs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-859) fuse-dfs utime behavior causes issues with tar
[ https://issues.apache.org/jira/browse/HDFS-859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HDFS-859: --- Resolution: Fixed Fix Version/s: 0.22.0 Assignee: Brian Bockelman Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I've just committed this. Thanks Brian! > fuse-dfs utime behavior causes issues with tar > -- > > Key: HDFS-859 > URL: https://issues.apache.org/jira/browse/HDFS-859 > Project: Hadoop HDFS > Issue Type: Bug > Components: contrib/fuse-dfs >Reporter: Brian Bockelman >Assignee: Brian Bockelman >Priority: Minor > Fix For: 0.22.0 > > Attachments: HDFS-859-2.patch, HDFS-859.patch > > > When trying to untar files onto fuse-dfs, tar will try to set the utime on > all the files and directories. However, setting the utime on a directory in > libhdfs causes an error. > We should silently ignore the failure of setting a utime on a directory; this > will allow tar to complete successfully. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-858) Incorrect return codes for fuse-dfs
[ https://issues.apache.org/jira/browse/HDFS-858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HDFS-858: --- Resolution: Fixed Fix Version/s: 0.22.0 Assignee: Brian Bockelman Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I've just committed this. Thanks Brian! (Test failures were unrelated.) > Incorrect return codes for fuse-dfs > --- > > Key: HDFS-858 > URL: https://issues.apache.org/jira/browse/HDFS-858 > Project: Hadoop HDFS > Issue Type: Bug > Components: contrib/fuse-dfs >Reporter: Brian Bockelman >Assignee: Brian Bockelman >Priority: Minor > Fix For: 0.22.0 > > Attachments: HDFS-858-2.patch, HDFS-858.patch > > > fuse-dfs doesn't pass proper error codes from libhdfs; places I'd like to > correct are hdfsFileOpen (which can result in permission denied or quota > violations) and hdfsWrite (which can result in quota violations). > By returning the correct error codes, command line utilities return much > better error messages - especially for quota violations, which can be a devil > to debug. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-857) Incorrect type for fuse-dfs capacity can cause "df" to return negative values on 32-bit machines
[ https://issues.apache.org/jira/browse/HDFS-857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HDFS-857: --- Resolution: Fixed Fix Version/s: 0.22.0 Assignee: Brian Bockelman Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I've just committed this. Thanks Brian! (Test failures were unrelated.) > Incorrect type for fuse-dfs capacity can cause "df" to return negative values > on 32-bit machines > > > Key: HDFS-857 > URL: https://issues.apache.org/jira/browse/HDFS-857 > Project: Hadoop HDFS > Issue Type: Bug > Components: contrib/fuse-dfs >Reporter: Brian Bockelman >Assignee: Brian Bockelman >Priority: Minor > Fix For: 0.22.0 > > Attachments: HDFS-857.patch > > > On sufficiently large HDFS installs, the casting of hdfsGetCapacity to a long > may cause "df" to return negative values. tOffset should be used instead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-856) Hardcoded replication level for new files in fuse-dfs
[ https://issues.apache.org/jira/browse/HDFS-856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HDFS-856: --- Resolution: Fixed Fix Version/s: 0.22.0 Assignee: Brian Bockelman Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I've just committed this. Thanks Brian! > Hardcoded replication level for new files in fuse-dfs > - > > Key: HDFS-856 > URL: https://issues.apache.org/jira/browse/HDFS-856 > Project: Hadoop HDFS > Issue Type: Bug > Components: contrib/fuse-dfs >Reporter: Brian Bockelman >Assignee: Brian Bockelman >Priority: Minor > Fix For: 0.22.0 > > Attachments: HADOOP-856.patch > > > In fuse-dfs, the number of replicas is always hardcoded to 3 in the arguments > to hdfsOpenFile. We should use the setting in the hadoop configuration > instead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-985) HDFS should issue multiple RPCs for listing a large directory
[ https://issues.apache.org/jira/browse/HDFS-985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-985: --- Attachment: iterativeLS_trunk.patch Here is the patch for the trunk. > HDFS should issue multiple RPCs for listing a large directory > - > > Key: HDFS-985 > URL: https://issues.apache.org/jira/browse/HDFS-985 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.22.0 > > Attachments: iterativeLS_trunk.patch, iterativeLS_yahoo.patch, > iterativeLS_yahoo1.patch, testFileStatus.patch > > > Currently HDFS issues one RPC from the client to the NameNode for listing a > directory. However some directories are large that contain thousands or > millions of items. Listing such large directories in one RPC has a few > shortcomings: > 1. The list operation holds the global fsnamesystem lock for a long time thus > blocking other requests. If a large number (like thousands) of such list > requests hit NameNode in a short period of time, NameNode will be > significantly slowed down. Users end up noticing longer response time or lost > connections to NameNode. > 2. The response message is uncontrollable big. We observed a response as big > as 50M bytes when listing a directory of 300 thousand items. Even with the > optimization introduced at HDFS-946 that may be able to cut the response by > 20-50%, the response size will still in the magnitude of 10 mega bytes. > I propose to implement a directory listing using multiple RPCs. Here is the > plan: > 1. Each getListing RPC has an upper limit on the number of items returned. > This limit could be configurable, but I am thinking to set it to be a fixed > number like 500. > 2. Each RPC additionally specifies a start position for this listing request. > I am thinking to use the last item of the previous listing RPC as an > indicator. Since NameNode stores all items in a directory as a sorted array, > NameNode uses the last item to locate the start item of this listing even if > the last item is deleted in between these two consecutive calls. This has the > advantage of avoid duplicate entries at the client side. > 3. The return value additionally specifies if the whole directory is done > listing. If the client sees a false flag, it will continue to issue another > RPC. > This proposal will change the semantics of large directory listing in a sense > that listing is no longer an atomic operation if a directory's content is > changing while the listing operation is in progress. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-985) HDFS should issue multiple RPCs for listing a large directory
[ https://issues.apache.org/jira/browse/HDFS-985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843694#action_12843694 ] Hairong Kuang commented on HDFS-985: I assume that you were talking about the comment line "// directory". I think this comment means that "now we are handling the case target is a directory." It has nothing to do with the statement "dfs.listPaths(target)" that I removed. > HDFS should issue multiple RPCs for listing a large directory > - > > Key: HDFS-985 > URL: https://issues.apache.org/jira/browse/HDFS-985 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.22.0 > > Attachments: iterativeLS_yahoo.patch, iterativeLS_yahoo1.patch, > testFileStatus.patch > > > Currently HDFS issues one RPC from the client to the NameNode for listing a > directory. However some directories are large that contain thousands or > millions of items. Listing such large directories in one RPC has a few > shortcomings: > 1. The list operation holds the global fsnamesystem lock for a long time thus > blocking other requests. If a large number (like thousands) of such list > requests hit NameNode in a short period of time, NameNode will be > significantly slowed down. Users end up noticing longer response time or lost > connections to NameNode. > 2. The response message is uncontrollable big. We observed a response as big > as 50M bytes when listing a directory of 300 thousand items. Even with the > optimization introduced at HDFS-946 that may be able to cut the response by > 20-50%, the response size will still in the magnitude of 10 mega bytes. > I propose to implement a directory listing using multiple RPCs. Here is the > plan: > 1. Each getListing RPC has an upper limit on the number of items returned. > This limit could be configurable, but I am thinking to set it to be a fixed > number like 500. > 2. Each RPC additionally specifies a start position for this listing request. > I am thinking to use the last item of the previous listing RPC as an > indicator. Since NameNode stores all items in a directory as a sorted array, > NameNode uses the last item to locate the start item of this listing even if > the last item is deleted in between these two consecutive calls. This has the > advantage of avoid duplicate entries at the client side. > 3. The return value additionally specifies if the whole directory is done > listing. If the client sees a false flag, it will continue to issue another > RPC. > This proposal will change the semantics of large directory listing in a sense > that listing is no longer an atomic operation if a directory's content is > changing while the listing operation is in progress. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads
[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843559#action_12843559 ] Zlatin Balevsky commented on HDFS-918: -- bq. Current DFS : 92MB/s over 60 runs bq. Multiplex : 97 MB/s over 60 runs bq. Either random variation, or maybe larger packet size helps A http://en.wikipedia.org/wiki/Student's_t-test will help you figure out if this difference is statistically significant or can be attributed to random variation. It is an essential tool when benchmarking modifications. The R project distro will make it trivial to perform. > Use single Selector and small thread pool to replace many instances of > BlockSender for reads > > > Key: HDFS-918 > URL: https://issues.apache.org/jira/browse/HDFS-918 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Reporter: Jay Booth > Fix For: 0.22.0 > > Attachments: hdfs-918-20100201.patch, hdfs-918-20100203.patch, > hdfs-918-20100211.patch, hdfs-918-20100228.patch, hdfs-918-20100309.patch, > hdfs-multiplex.patch > > > Currently, on read requests, the DataXCeiver server allocates a new thread > per request, which must allocate its own buffers and leads to > higher-than-optimal CPU and memory usage by the sending threads. If we had a > single selector and a small threadpool to multiplex request packets, we could > theoretically achieve higher performance while taking up fewer resources and > leaving more CPU on datanodes available for mapred, hbase or whatever. This > can be done without changing any wire protocols. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1007) HFTP needs to be updated to use delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-1007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-1007: -- Attachment: distcp-hftp-2.1.1.patch Patch adding a bugfix in the last patch that Boris attached. > HFTP needs to be updated to use delegation tokens > - > > Key: HDFS-1007 > URL: https://issues.apache.org/jira/browse/HDFS-1007 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 0.22.0 >Reporter: Devaraj Das > Fix For: 0.22.0 > > Attachments: distcp-hftp-2.1.1.patch, distcp-hftp.1.patch, > distcp-hftp.2.1.patch, distcp-hftp.2.patch, distcp-hftp.patch, > HDFS-1007-BP20.patch > > > HFTPFileSystem should be updated to use the delegation tokens so that it can > talk to the secure namenodes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
