[jira] Updated: (HDFS-703) Replace current fault injection implementation with one from Common
[ https://issues.apache.org/jira/browse/HDFS-703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-703: Attachment: HDFS-703.patch Initial patch > Replace current fault injection implementation with one from Common > --- > > Key: HDFS-703 > URL: https://issues.apache.org/jira/browse/HDFS-703 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.22.0 >Reporter: Konstantin Boudnik >Assignee: Konstantin Boudnik > Attachments: HDFS-703.patch > > > After HADOOP-6204 has been implemented HDFS doesn't need to have its separate > implementation of fault injection framework. Instead it has to reuse one from > Common. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HDFS-732) HDFS files are ending up truncated
[ https://issues.apache.org/jira/browse/HDFS-732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE resolved HDFS-732. - Resolution: Fixed Assignee: Tsz Wo (Nicholas), SZE I have committed this to 0.20 only. > HDFS files are ending up truncated > -- > > Key: HDFS-732 > URL: https://issues.apache.org/jira/browse/HDFS-732 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs client >Affects Versions: 0.20.2 >Reporter: Christian Kunz >Assignee: Tsz Wo (Nicholas), SZE >Priority: Blocker > Fix For: 0.20.2 > > Attachments: h732_20091028_0.20.patch > > > We recently started to use hadoop-0.20.1 in our production environment (less > than 2 weeks ago) and already had 3 instances of truncated files, more than > we had for months using hadoop-0.18.3. > Writing is done using libhdfs, although it rather seems to be a problem on > the server side. > I will post some relevant logs (they are too large to be put into the > description) > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-732) HDFS files are ending up truncated
[
https://issues.apache.org/jira/browse/HDFS-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771739#action_12771739
]
Tsz Wo (Nicholas), SZE commented on HDFS-732:
-
Tested on 0.20:
{noformat}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new
or modified tests.
[exec] Please justify why no tests are needed for
this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath
integrity.
{noformat}
All unit tests passed except TestDatanodeBlockScanner, TestFsck and
TestReduceFetch. The failures are not related to the patch. These three tests
also failed on a clean 0.20 trunk in my machine. See HDFS-734 for
TestDatanodeBlockScanner. I will file new issues for TestFsck and
TestReduceFetch.
No new test is added since the change is very simple.
I am going to commit the patch to 0.20 only since we don't have plan to release
0.19.3.
> HDFS files are ending up truncated
> --
>
> Key: HDFS-732
> URL: https://issues.apache.org/jira/browse/HDFS-732
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Affects Versions: 0.20.2
>Reporter: Christian Kunz
>Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: h732_20091028_0.20.patch
>
>
> We recently started to use hadoop-0.20.1 in our production environment (less
> than 2 weeks ago) and already had 3 instances of truncated files, more than
> we had for months using hadoop-0.18.3.
> Writing is done using libhdfs, although it rather seems to be a problem on
> the server side.
> I will post some relevant logs (they are too large to be put into the
> description)
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-736) commitBlockSynchronization() should directly update block GS and length.
[
https://issues.apache.org/jira/browse/HDFS-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771732#action_12771732
]
Hudson commented on HDFS-736:
-
Integrated in Hadoop-Hdfs-trunk-Commit #90 (See
[http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/90/])
. commitBlockSynchronization() updates block GS and length in-place.
Contributed by Konstantin Shvachko.
> commitBlockSynchronization() should directly update block GS and length.
>
>
> Key: HDFS-736
> URL: https://issues.apache.org/jira/browse/HDFS-736
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
>Affects Versions: 0.21.0
>Reporter: Konstantin Shvachko
>Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: commitBlockSync.patch
>
>
> {{FSNamesystem.commitBlockSynchronization()}} updates block's generation
> stamp and length by first removing the old block instance from blocksMap and
> then inserting the new instance back. This was necessary when GS was a part
> of the block key. After HDFS-512 the GS along with the block length can be
> updated directly in the blocksMap entry.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-731) Support new Syncable interface in HDFS
[ https://issues.apache.org/jira/browse/HDFS-731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771730#action_12771730 ] Suresh Srinivas commented on HDFS-731: -- Currently hsync and hflush are the same. We should create a jira to track completing hsync implementation. That jira should be linked to this. +1 for the patch. > Support new Syncable interface in HDFS > -- > > Key: HDFS-731 > URL: https://issues.apache.org/jira/browse/HDFS-731 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.21.0 >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.21.0 > > Attachments: hdfsSyncable.patch, hdfsSyncable1.patch, > hdfsSyncable2.patch > > > HDFS should implement the new Syncable interface defined in HADOOP-6313. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-736) commitBlockSynchronization() should directly update block GS and length.
[
https://issues.apache.org/jira/browse/HDFS-736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HDFS-736:
-
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this.
> commitBlockSynchronization() should directly update block GS and length.
>
>
> Key: HDFS-736
> URL: https://issues.apache.org/jira/browse/HDFS-736
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
>Affects Versions: 0.21.0
>Reporter: Konstantin Shvachko
>Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: commitBlockSync.patch
>
>
> {{FSNamesystem.commitBlockSynchronization()}} updates block's generation
> stamp and length by first removing the old block instance from blocksMap and
> then inserting the new instance back. This was necessary when GS was a part
> of the block key. After HDFS-512 the GS along with the block length can be
> updated directly in the blocksMap entry.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-736) commitBlockSynchronization() should directly update block GS and length.
[
https://issues.apache.org/jira/browse/HDFS-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771721#action_12771721
]
Konstantin Shvachko commented on HDFS-736:
--
No new tests required, modified functionality is checked by the existing tests
TestLeaseRecovery and TestLeaseRecovery2.
> commitBlockSynchronization() should directly update block GS and length.
>
>
> Key: HDFS-736
> URL: https://issues.apache.org/jira/browse/HDFS-736
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
>Affects Versions: 0.21.0
>Reporter: Konstantin Shvachko
>Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: commitBlockSync.patch
>
>
> {{FSNamesystem.commitBlockSynchronization()}} updates block's generation
> stamp and length by first removing the old block instance from blocksMap and
> then inserting the new instance back. This was necessary when GS was a part
> of the block key. After HDFS-512 the GS along with the block length can be
> updated directly in the blocksMap entry.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-731) Support new Syncable interface in HDFS
[ https://issues.apache.org/jira/browse/HDFS-731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771716#action_12771716 ] Hairong Kuang commented on HDFS-731: Since this patch needs the changes made in HDFS-6313, I used a modified version of test-patch.sh to "ant test-patch". Here is the result: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 33 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. > Support new Syncable interface in HDFS > -- > > Key: HDFS-731 > URL: https://issues.apache.org/jira/browse/HDFS-731 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.21.0 >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.21.0 > > Attachments: hdfsSyncable.patch, hdfsSyncable1.patch, > hdfsSyncable2.patch > > > HDFS should implement the new Syncable interface defined in HADOOP-6313. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[
https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771715#action_12771715
]
dhruba borthakur commented on HDFS-729:
---
I am planning to follow Raghu's advice and add the following API to the
namenode:
{quote}
/**
* Returns a list of files that are corrupted.
*
* Returns a list of files that have at least one block that has no valid
replicas.
* The returned list has numExpectedFiles files in it. If the number of files
* returned is smaller than numExpectedFiles, then it implies that no more
* corrupted files are available in the system. The startingNumber is the
* startingNumber-th corrupted file in the system.
*
* @param numExpectedFiles the maximum number of files to be returned
* @param startingNumber list files starting from startingNumberth to
* (startingNumber + numExpectedFiles)th in the
* list of corrupted files
* @throws AccessControlException if the superuser privilege is violated.
* @throws IOException if unable to retrieve information of a corrupt file
*/
public LocatedBlocks[] getCorruptFiles(int numExpectedFiles, int
startingNumber) throws IOException;
{quote}
This will be used by fsck (or any other application) to quickly detect
corrupted files.
> fsck option to list only corrupted files
>
>
> Key: HDFS-729
> URL: https://issues.apache.org/jira/browse/HDFS-729
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
>
> An option to fsck to list only corrupted files will be very helpful for
> frequent monitoring.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-731) Support new Syncable interface in HDFS
[ https://issues.apache.org/jira/browse/HDFS-731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-731: --- Attachment: hdfsSyncable2.patch This patch sync with the trunk. > Support new Syncable interface in HDFS > -- > > Key: HDFS-731 > URL: https://issues.apache.org/jira/browse/HDFS-731 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.21.0 >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.21.0 > > Attachments: hdfsSyncable.patch, hdfsSyncable1.patch, > hdfsSyncable2.patch > > > HDFS should implement the new Syncable interface defined in HADOOP-6313. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-737) Improvement in metasave output
[ https://issues.apache.org/jira/browse/HDFS-737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771690#action_12771690 ] Hudson commented on HDFS-737: - Integrated in Hadoop-Hdfs-trunk-Commit #89 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/89/]) . Add full path name of the file to the block information and summary of total number of files, blocks, live and deadnodes to metasave output. Contributed by Jitendra Pandey. > Improvement in metasave output > -- > > Key: HDFS-737 > URL: https://issues.apache.org/jira/browse/HDFS-737 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Fix For: 0.21.0, 0.22.0 > > Attachments: HDFS-737.1.patch, HDFS-737.2.patch, HDFS-737.3.patch > > > This jira tracks following improvements in metasave output > 1. A summary of total files/directories and blocks at the begining > 2. Full path name of the files should also be written out for > under-replicated/corrupt or missing blocks. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-732) HDFS files are ending up truncated
[ https://issues.apache.org/jira/browse/HDFS-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771675#action_12771675 ] dhruba borthakur commented on HDFS-732: --- > Should we brother fixing 0.19? Either way is fine. I will have to pull it into pur 0.19 hadoop cluster as well. > HDFS files are ending up truncated > -- > > Key: HDFS-732 > URL: https://issues.apache.org/jira/browse/HDFS-732 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs client >Affects Versions: 0.20.2 >Reporter: Christian Kunz >Priority: Blocker > Fix For: 0.20.2 > > Attachments: h732_20091028_0.20.patch > > > We recently started to use hadoop-0.20.1 in our production environment (less > than 2 weeks ago) and already had 3 instances of truncated files, more than > we had for months using hadoop-0.18.3. > Writing is done using libhdfs, although it rather seems to be a problem on > the server side. > I will post some relevant logs (they are too large to be put into the > description) > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-737) Improvement in metasave output
[ https://issues.apache.org/jira/browse/HDFS-737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-737: - Component/s: name-node > Improvement in metasave output > -- > > Key: HDFS-737 > URL: https://issues.apache.org/jira/browse/HDFS-737 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Fix For: 0.21.0, 0.22.0 > > Attachments: HDFS-737.1.patch, HDFS-737.2.patch, HDFS-737.3.patch > > > This jira tracks following improvements in metasave output > 1. A summary of total files/directories and blocks at the begining > 2. Full path name of the files should also be written out for > under-replicated/corrupt or missing blocks. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-737) Improvement in metasave output
[ https://issues.apache.org/jira/browse/HDFS-737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-737: - Resolution: Fixed Fix Version/s: 0.21.0 Release Note: Add full path name of the file to the block information and summary of total number of files, blocks, live and dead datanodes to metasave output. Hadoop Flags: [Incompatible change, Reviewed] Status: Resolved (was: Patch Available) Committed the patch to both trunk and 21. > Improvement in metasave output > -- > > Key: HDFS-737 > URL: https://issues.apache.org/jira/browse/HDFS-737 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Fix For: 0.21.0, 0.22.0 > > Attachments: HDFS-737.1.patch, HDFS-737.2.patch, HDFS-737.3.patch > > > This jira tracks following improvements in metasave output > 1. A summary of total files/directories and blocks at the begining > 2. Full path name of the files should also be written out for > under-replicated/corrupt or missing blocks. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-736) commitBlockSynchronization() should directly update block GS and length.
[
https://issues.apache.org/jira/browse/HDFS-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771630#action_12771630
]
Jitendra Nath Pandey commented on HDFS-736:
---
+1 for the patch.
> commitBlockSynchronization() should directly update block GS and length.
>
>
> Key: HDFS-736
> URL: https://issues.apache.org/jira/browse/HDFS-736
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
>Affects Versions: 0.21.0
>Reporter: Konstantin Shvachko
>Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: commitBlockSync.patch
>
>
> {{FSNamesystem.commitBlockSynchronization()}} updates block's generation
> stamp and length by first removing the old block instance from blocksMap and
> then inserting the new instance back. This was necessary when GS was a part
> of the block key. After HDFS-512 the GS along with the block length can be
> updated directly in the blocksMap entry.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-521) Create new tests for pipeline
[
https://issues.apache.org/jira/browse/HDFS-521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Boudnik updated HDFS-521:
Attachment: HDFS-521.patch
Addressing Hairong's comments:
- datanodes order is now preserved by using a List instead of HashMap
- the invariant is verified across all datanodes
- redundant wait is removed
- testcase _03 is suppose to be covered by {{TestReadWhileWriting}} however a
multiple packet writes creates some strange problems in the test, which might
be taken care off separately
> Create new tests for pipeline
> -
>
> Key: HDFS-521
> URL: https://issues.apache.org/jira/browse/HDFS-521
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
>Affects Versions: 0.21.0, 0.22.0
>Reporter: Konstantin Boudnik
>Assignee: Konstantin Boudnik
> Attachments: HDFS-521.patch, HDFS-521.patch, HDFS-521.patch,
> HDFS-521.patch, HDFS-521.patch, HDFS-521.patch, HDFS-521.patch,
> HDFS-521.patch, HDFS-521.patch, HDFS-521.patch, HDFS-521.patch,
> hdfs-521.patch, hdfs-521.patch, hdfs-521.patch, HDFS-521.patch,
> HDFS-521.patch, HDFS-521.patch
>
>
> This JIRA is created to be a place holder for the test implemented as a part
> of HDFS-483 fix
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-732) HDFS files are ending up truncated
[ https://issues.apache.org/jira/browse/HDFS-732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-732: Priority: Blocker (was: Major) Affects Version/s: (was: 0.20.1) 0.20.2 Fix Version/s: 0.20.2 Hadoop Flags: [Reviewed] > Hi nicholas, can we put this patch in 0.20.3 release? Thanks. I believe this is qualified to be a blocker of 0.20. Should we brother fixing 0.19? > HDFS files are ending up truncated > -- > > Key: HDFS-732 > URL: https://issues.apache.org/jira/browse/HDFS-732 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs client >Affects Versions: 0.20.2 >Reporter: Christian Kunz >Priority: Blocker > Fix For: 0.20.2 > > Attachments: h732_20091028_0.20.patch > > > We recently started to use hadoop-0.20.1 in our production environment (less > than 2 weeks ago) and already had 3 instances of truncated files, more than > we had for months using hadoop-0.18.3. > Writing is done using libhdfs, although it rather seems to be a problem on > the server side. > I will post some relevant logs (they are too large to be put into the > description) > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-737) Improvement in metasave output
[ https://issues.apache.org/jira/browse/HDFS-737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771587#action_12771587 ] Jitendra Nath Pandey commented on HDFS-737: --- The failure of TestBlockReport is not related to this patch. > Improvement in metasave output > -- > > Key: HDFS-737 > URL: https://issues.apache.org/jira/browse/HDFS-737 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Fix For: 0.22.0 > > Attachments: HDFS-737.1.patch, HDFS-737.2.patch, HDFS-737.3.patch > > > This jira tracks following improvements in metasave output > 1. A summary of total files/directories and blocks at the begining > 2. Full path name of the files should also be written out for > under-replicated/corrupt or missing blocks. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-632) hdfsJniHelper.c: #include is not portable
[ https://issues.apache.org/jira/browse/HDFS-632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771509#action_12771509 ] Hadoop QA commented on HDFS-632: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423597/HDFS-632.patch against trunk revision 830804. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/89/console This message is automatically generated. > hdfsJniHelper.c: #include is not portable > --- > > Key: HDFS-632 > URL: https://issues.apache.org/jira/browse/HDFS-632 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Allen Wittenauer > Attachments: HDFS-632.patch > > > hdfsJniHelper.c includes but this appears to be unnecessary, since > even under Linux none of the routines that are prototyped are used. Worse > yet, error.h doesn't appear to be a standard header file so this breaks on > Mac OS X and Solaris and prevents libhdfs from being built. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-632) hdfsJniHelper.c: #include is not portable
[ https://issues.apache.org/jira/browse/HDFS-632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771494#action_12771494 ] Allen Wittenauer commented on HDFS-632: --- (this should probably be in MapReduce, since that is where the code is at. Very confusing.) :( > hdfsJniHelper.c: #include is not portable > --- > > Key: HDFS-632 > URL: https://issues.apache.org/jira/browse/HDFS-632 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Allen Wittenauer > Attachments: HDFS-632.patch > > > hdfsJniHelper.c includes but this appears to be unnecessary, since > even under Linux none of the routines that are prototyped are used. Worse > yet, error.h doesn't appear to be a standard header file so this breaks on > Mac OS X and Solaris and prevents libhdfs from being built. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-632) hdfsJniHelper.c: #include is not portable
[ https://issues.apache.org/jira/browse/HDFS-632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-632: -- Status: Patch Available (was: Open) > hdfsJniHelper.c: #include is not portable > --- > > Key: HDFS-632 > URL: https://issues.apache.org/jira/browse/HDFS-632 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Allen Wittenauer > Attachments: HDFS-632.patch > > > hdfsJniHelper.c includes but this appears to be unnecessary, since > even under Linux none of the routines that are prototyped are used. Worse > yet, error.h doesn't appear to be a standard header file so this breaks on > Mac OS X and Solaris and prevents libhdfs from being built. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-632) hdfsJniHelper.c: #include is not portable
[ https://issues.apache.org/jira/browse/HDFS-632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-632: -- Attachment: HDFS-632.patch Removes error.h from hdfsJniHelper.c, which doesn't appear to be necessary and breaks compatibility on several operating systems. > hdfsJniHelper.c: #include is not portable > --- > > Key: HDFS-632 > URL: https://issues.apache.org/jira/browse/HDFS-632 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Allen Wittenauer > Attachments: HDFS-632.patch > > > hdfsJniHelper.c includes but this appears to be unnecessary, since > even under Linux none of the routines that are prototyped are used. Worse > yet, error.h doesn't appear to be a standard header file so this breaks on > Mac OS X and Solaris and prevents libhdfs from being built. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-741) TestHFlush test doesn't seek() past previously written part of the file
[
https://issues.apache.org/jira/browse/HDFS-741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771439#action_12771439
]
Konstantin Boudnik commented on HDFS-741:
-
I have verified it: the whole file reads Ok. I've simply commented out
{{checkData(..)}} call inside of the loop and tests start passing. Seems to me
as a {{seek()}}} problem.
> TestHFlush test doesn't seek() past previously written part of the file
> ---
>
> Key: HDFS-741
> URL: https://issues.apache.org/jira/browse/HDFS-741
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 0.21.0, 0.22.0
>Reporter: Konstantin Boudnik
>Assignee: Konstantin Boudnik
> Attachments: HDFS-741.patch, HDFS-741.patch
>
>
> As a part of the test scenario a file is being written, 10th of the total
> length in a time. Then a reader is opened to read what has been just written
> and hflush'ed. However, it always starts reading from the 0 position of the
> file and doesn't seek to the start of the portion written last.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-691) Limitation on java.io.InputStream.available()
[ https://issues.apache.org/jira/browse/HDFS-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771419#action_12771419 ] Hudson commented on HDFS-691: - Integrated in Hadoop-Hdfs-trunk #124 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/124/]) . Fix an overflow error in DFSClient.DFSInputStream.available(). > Limitation on java.io.InputStream.available() > - > > Key: HDFS-691 > URL: https://issues.apache.org/jira/browse/HDFS-691 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs client >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 0.21.0, 0.22.0 > > Attachments: h691_20091028.patch > > > java.io.InputStream.available() returns an int which has the max value 2^31-1 > = 2GB - 1B. It won't work if the number of available bytes is >= 2GB. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-222) Support for concatenating of files into a single file
[ https://issues.apache.org/jira/browse/HDFS-222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771418#action_12771418 ] Hudson commented on HDFS-222: - Integrated in Hadoop-Hdfs-trunk #124 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/124/]) . Support for concatenating of files into a single file without copying. Contributed by Boris Shkolnik. > Support for concatenating of files into a single file > - > > Key: HDFS-222 > URL: https://issues.apache.org/jira/browse/HDFS-222 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Venkatesh S >Assignee: Boris Shkolnik > Fix For: 0.22.0 > > Attachments: HDFS-222-1.patch, HDFS-222-10.patch, HDFS-222-10.patch, > HDFS-222-2.patch, HDFS-222-3.patch, HDFS-222-4.patch, HDFS-222-5.patch, > HDFS-222-6.patch, HDFS-222-7.patch, HDFS-222-8.patch, HDFS-222-9.patch, > HDFS-222-9.patch, HDFS-222.patch > > > An API to concatenate files of same size and replication factor on HDFS into > a single larger file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-743) file size is fluctuating although file is closed
[ https://issues.apache.org/jira/browse/HDFS-743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-743: -- Attachment: fluctuatingFileSize_0.17.txt This patch is for 0.17 > file size is fluctuating although file is closed > > > Key: HDFS-743 > URL: https://issues.apache.org/jira/browse/HDFS-743 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Reporter: dhruba borthakur >Assignee: dhruba borthakur >Priority: Blocker > Attachments: fluctuatingFileSize_0.17.txt > > > I am seeing that the length of a file sometimes becomes zero after a namenode > restart. These files have only one block. All the three replicas of that > block on the datanode(s) has non-zero size. Increasing the replication factor > of the file causes the file to show its correct non-zero length. > I am marking this as a blocker because it is still to be investigated which > releases it affects. I am seeing this on 0.17.x very frequently. I might have > seen this on 0.20.x but do not have a reproducible case yet. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-743) file size is fluctuating although file is closed
[ https://issues.apache.org/jira/browse/HDFS-743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771326#action_12771326 ] dhruba borthakur commented on HDFS-743: --- An application wrote 200 bytes to a file but failed to close the file. The replicas all have 200 bytes of the file. The namenode still thinks that the filesize is zero bytes. When the hardlease expires, the namenode writes the OP_CLOSE record to the transaction log, the filelength for this block recorded in the transaction log is zero. This by itself is not a bug. In hadoop 0.17, the namenode does not contact datanodes while doing lease recovery. If the namenode is rebooted, the filesize continues to be displayed as zero. The block report from the datanodes sends the blocksize to be 200 bytes. But because of a bug in DatanodeDescriptor.reportDiff, this block size of 200 is ignored by the namenode. This, by itself is still not a bug because the file continues to be displayed as having zero size. Now, if a datanode dies, the block of size 200 is replicated from one of the original datanodes to a new datanode. The new datanode sends a blockReceived command. The blockReceived command has the size 200 and the namenode accepts this value as the true size of the block and updates the block length to be 200 bytes. The file is now displayed as having 200 bytes. If one restarts the namenode, the file goes back to being zero size until the time when the block needs to be replicated again and the file shows up being of size 200. This is the cause of the mystery of the "fluctuating file size". > file size is fluctuating although file is closed > > > Key: HDFS-743 > URL: https://issues.apache.org/jira/browse/HDFS-743 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Reporter: dhruba borthakur >Assignee: dhruba borthakur >Priority: Blocker > > I am seeing that the length of a file sometimes becomes zero after a namenode > restart. These files have only one block. All the three replicas of that > block on the datanode(s) has non-zero size. Increasing the replication factor > of the file causes the file to show its correct non-zero length. > I am marking this as a blocker because it is still to be investigated which > releases it affects. I am seeing this on 0.17.x very frequently. I might have > seen this on 0.20.x but do not have a reproducible case yet. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-743) file size is fluctuating although file is closed
file size is fluctuating although file is closed Key: HDFS-743 URL: https://issues.apache.org/jira/browse/HDFS-743 Project: Hadoop HDFS Issue Type: Bug Components: name-node Reporter: dhruba borthakur Assignee: dhruba borthakur Priority: Blocker I am seeing that the length of a file sometimes becomes zero after a namenode restart. These files have only one block. All the three replicas of that block on the datanode(s) has non-zero size. Increasing the replication factor of the file causes the file to show its correct non-zero length. I am marking this as a blocker because it is still to be investigated which releases it affects. I am seeing this on 0.17.x very frequently. I might have seen this on 0.20.x but do not have a reproducible case yet. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-611) Heartbeats times from Datanodes increase when there are plenty of blocks to delete
[ https://issues.apache.org/jira/browse/HDFS-611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771311#action_12771311 ] Hadoop QA commented on HDFS-611: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423525/HDFS-611.trunk.v2.patch against trunk revision 830804. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/88/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/88/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/88/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/88/console This message is automatically generated. > Heartbeats times from Datanodes increase when there are plenty of blocks to > delete > -- > > Key: HDFS-611 > URL: https://issues.apache.org/jira/browse/HDFS-611 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 0.20.1, 0.21.0, 0.22.0 >Reporter: dhruba borthakur >Assignee: Zheng Shao > Fix For: 0.20.2, 0.21.0, 0.22.0 > > Attachments: HDFS-611.branch-19.patch, HDFS-611.branch-19.v2.patch, > HDFS-611.branch-20.patch, HDFS-611.branch-20.v2.patch, HDFS-611.trunk.patch, > HDFS-611.trunk.v2.patch > > > I am seeing that when we delete a large directory that has plenty of blocks, > the heartbeat times from datanodes increase significantly from the normal > value of 3 seconds to as large as 50 seconds or so. The heartbeat thread in > the Datanode deletes a bunch of blocks sequentially, this causes the > heartbeat times to increase. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
