[jira] Commented: (HDFS-1024) SecondaryNamenode fails to checkpoint because namenode fails with CancelledKeyException

[ https://issues.apache.org/jira/browse/HDFS-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12844607#action_12844607 ] dhruba borthakur commented on HDFS-1024: +1 code looks good. > SecondaryNamenode fa

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12844185#action_12844185 ] dhruba borthakur commented on HDFS-918: --- Thanks for the description Jay. What is ur fee

[jira] Created: (HDFS-1034) Enhance datanode to read data and checksum file in parallel

Enhance datanode to read data and checksum file in parallel --- Key: HDFS-1034 URL: https://issues.apache.org/jira/browse/HDFS-1034 Project: Hadoop HDFS Issue Type: Improvement

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12844160#action_12844160 ] dhruba borthakur commented on HDFS-918: --- Thsi patch uses nio to multiplex IO over socke

[jira] Commented: (HDFS-854) Datanode should scan devices in parallel to generate block report

[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843935#action_12843935 ] dhruba borthakur commented on HDFS-854: --- > Or probably keep last added/deleted blocks i

[jira] Updated: (HDFS-850) Display more memory details on the web ui

[ https://issues.apache.org/jira/browse/HDFS-850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-850: -- Resolution: Fixed Fix Version/s: 0.22.0 Hadoop Flags: [Reviewed] Status: Re

[jira] Commented: (HDFS-850) Display more memory details on the web ui

[ https://issues.apache.org/jira/browse/HDFS-850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843927#action_12843927 ] dhruba borthakur commented on HDFS-850: --- +1 Code looks good. I will commit it. > Displ

[jira] Commented: (HDFS-826) Allow a mechanism for an application to detect that datanode(s) have died in the write pipeline

[ https://issues.apache.org/jira/browse/HDFS-826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842946#action_12842946 ] dhruba borthakur commented on HDFS-826: --- Thanks stack for reviewing this one. I will co

[jira] Updated: (HDFS-729) fsck option to list only corrupted files

[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-729: -- Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available

[jira] Created: (HDFS-1031) Enhance the webUi to list a few of the corrupted files in HDFS

Enhance the webUi to list a few of the corrupted files in HDFS -- Key: HDFS-1031 URL: https://issues.apache.org/jira/browse/HDFS-1031 Project: Hadoop HDFS Issue Type: Improvement

[jira] Commented: (HDFS-1028) INode.getPathNames could split more efficiently

[ https://issues.apache.org/jira/browse/HDFS-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842464#action_12842464 ] dhruba borthakur commented on HDFS-1028: +1 seems like a good idea. > INode.getPath

[jira] Commented: (HDFS-729) fsck option to list only corrupted files

[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842362#action_12842362 ] dhruba borthakur commented on HDFS-729: --- Thanks Rodrigo, I will commit this patch. > f

[jira] Commented: (HDFS-1024) SecondaryNamenode fails to checkpoint because namenode fails with CancelledKeyException

[ https://issues.apache.org/jira/browse/HDFS-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842041#action_12842041 ] dhruba borthakur commented on HDFS-1024: One proposal to fix this issue is that the

[jira] Commented: (HDFS-1024) SecondaryNamenode fails to checkpoint because namenode fails with CancelledKeyException

[ https://issues.apache.org/jira/browse/HDFS-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842036#action_12842036 ] dhruba borthakur commented on HDFS-1024: The namenode logs show the following: {cod

[jira] Commented: (HDFS-1024) SecondaryNamenode fails to checkpoint because namenode fails with CancelledKeyException

[ https://issues.apache.org/jira/browse/HDFS-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842037#action_12842037 ] dhruba borthakur commented on HDFS-1024: In the above case, the size of the actual f

[jira] Created: (HDFS-1024) SecondaryNamenode fails to checkpoint because namenode fails with CancelledKeyException

SecondaryNamenode fails to checkpoint because namenode fails with CancelledKeyException --- Key: HDFS-1024 URL: https://issues.apache.org/jira/browse/HDFS-1024 Projec

[jira] Commented: (HDFS-826) Allow a mechanism for an application to detect that datanode(s) have died in the write pipeline

[ https://issues.apache.org/jira/browse/HDFS-826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841754#action_12841754 ] dhruba borthakur commented on HDFS-826: --- Hi Hairong, Thanks for the review. I would rea

[jira] Updated: (HDFS-966) NameNode recovers lease even in safemode

[ https://issues.apache.org/jira/browse/HDFS-966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-966: -- Attachment: leaseRecoverSafeMode.txt > NameNode recovers lease even in safemode > ---

[jira] Commented: (HDFS-826) Allow a mechanism for an application to detect that datanode(s) have died in the write pipeline

[ https://issues.apache.org/jira/browse/HDFS-826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841494#action_12841494 ] dhruba borthakur commented on HDFS-826: --- Hauirong: it makes the application code easier

[jira] Commented: (HDFS-826) Allow a mechanism for an application to detect that datanode(s) have died in the write pipeline

[ https://issues.apache.org/jira/browse/HDFS-826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841021#action_12841021 ] dhruba borthakur commented on HDFS-826: --- anybody wants to re-review this one? Thanks.

[jira] Commented: (HDFS-729) fsck option to list only corrupted files

[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840804#action_12840804 ] dhruba borthakur commented on HDFS-729: --- +1 Code looks good. > fsck option to list onl

[jira] Commented: (HDFS-867) Add a PowerTopology class to aid replica placement and enhance availability of blocks

[ https://issues.apache.org/jira/browse/HDFS-867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840522#action_12840522 ] dhruba borthakur commented on HDFS-867: --- can you use HDFS-385? > Add a PowerTopology c

[jira] Updated: (HDFS-826) Allow a mechanism for an application to detect that datanode(s) have died in the write pipeline

[ https://issues.apache.org/jira/browse/HDFS-826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-826: -- Status: Patch Available (was: Open) > Allow a mechanism for an application to detect that datano

[jira] Updated: (HDFS-826) Allow a mechanism for an application to detect that datanode(s) have died in the write pipeline

[ https://issues.apache.org/jira/browse/HDFS-826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-826: -- Attachment: Replicable4.txt 1. Removed replicable interface 2. If it is in-between blocks and the

[jira] Updated: (HDFS-826) Allow a mechanism for an application to detect that datanode(s) have died in the write pipeline

[ https://issues.apache.org/jira/browse/HDFS-826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-826: -- Status: Open (was: Patch Available) Cancelling to incorporate review comments. > Allow a mechan

[jira] Commented: (HDFS-729) fsck option to list only corrupted files

[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840378#action_12840378 ] dhruba borthakur commented on HDFS-729: --- Code looks good. The only question I have is t

[jira] Updated: (HDFS-988) saveNamespace can corrupt edits log

[ https://issues.apache.org/jira/browse/HDFS-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-988: -- Tags: hbase > saveNamespace can corrupt edits log > --- > >

[jira] Commented: (HDFS-826) Allow a mechanism for an application to detect that datanode(s) have died in the write pipeline

[ https://issues.apache.org/jira/browse/HDFS-826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839853#action_12839853 ] dhruba borthakur commented on HDFS-826: --- Thanks Todd for the review. I agree with your

[jira] Updated: (HDFS-826) Allow a mechanism for an application to detect that datanode(s) have died in the write pipeline

[ https://issues.apache.org/jira/browse/HDFS-826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-826: -- Status: Patch Available (was: Open) Can somebody pl review this patch? This is needed to make HB

[jira] Updated: (HDFS-729) fsck option to list only corrupted files

[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-729: -- Status: Open (was: Patch Available) Thanks Rodrigo. I will wait for your new patch. > fsck opti

[jira] Commented: (HDFS-870) Topology is permanently cached

[ https://issues.apache.org/jira/browse/HDFS-870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12838608#action_12838608 ] dhruba borthakur commented on HDFS-870: --- Hi Koji, we do not use a external script to ma

[jira] Commented: (HDFS-686) NullPointerException is thrown while merging edit log and image

[ https://issues.apache.org/jira/browse/HDFS-686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12838550#action_12838550 ] dhruba borthakur commented on HDFS-686: --- Ryan/Andrew: are you both seeing this on hadoo

[jira] Commented: (HDFS-729) fsck option to list only corrupted files

[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837072#action_12837072 ] dhruba borthakur commented on HDFS-729: --- Oh, and maybe the unit test can check that no

[jira] Commented: (HDFS-729) fsck option to list only corrupted files

[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837071#action_12837071 ] dhruba borthakur commented on HDFS-729: --- Code looks good. A few comments: 1. TestDFSCl

[jira] Commented: (HDFS-708) A stress-test tool for HDFS.

[ https://issues.apache.org/jira/browse/HDFS-708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12836494#action_12836494 ] dhruba borthakur commented on HDFS-708: --- Thanks for the document. a few questions: 1.

[jira] Resolved: (HDFS-87) NameNode startup fails if edit log terminates prematurely

[ https://issues.apache.org/jira/browse/HDFS-87?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur resolved HDFS-87. -- Resolution: Not A Problem > NameNode startup fails if edit log terminates prematurely > --

[jira] Commented: (HDFS-87) NameNode startup fails if edit log terminates prematurely

[ https://issues.apache.org/jira/browse/HDFS-87?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12835945#action_12835945 ] dhruba borthakur commented on HDFS-87: -- The issue of having an truncated last-entry in th

[jira] Updated: (HDFS-988) saveNamespace can corrupt edits log

[ https://issues.apache.org/jira/browse/HDFS-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-988: -- Attachment: saveNamespace.txt This patch is for hadoop 0.20. It fixes the race of saveNamespace w

[jira] Commented: (HDFS-686) NullPointerException is thrown while merging edit log and image

[ https://issues.apache.org/jira/browse/HDFS-686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12835585#action_12835585 ] dhruba borthakur commented on HDFS-686: --- I see that this patch was already committed to

[jira] Commented: (HDFS-988) saveNamespace can corrupt edits log

[ https://issues.apache.org/jira/browse/HDFS-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12835486#action_12835486 ] dhruba borthakur commented on HDFS-988: --- I agree that HDFS-956 could be fixed. But HDFS

[jira] Commented: (HDFS-909) Race condition between rollEditLog or rollFSImage ant FSEditsLog.write operations corrupts edits log

[ https://issues.apache.org/jira/browse/HDFS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12835464#action_12835464 ] dhruba borthakur commented on HDFS-909: --- I also posted a patch just fr the savenamespac

[jira] Commented: (HDFS-988) saveNamespace can corrupt edits log

[ https://issues.apache.org/jira/browse/HDFS-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12835406#action_12835406 ] dhruba borthakur commented on HDFS-988: --- I think the wait time is theoretically unbound

[jira] Commented: (HDFS-988) saveNamespace can corrupt edits log

[ https://issues.apache.org/jira/browse/HDFS-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12835403#action_12835403 ] dhruba borthakur commented on HDFS-988: --- My proposal is to make the enterSafeMode metho

[jira] Created: (HDFS-988) saveNamespace can corrupt edits log

saveNamespace can corrupt edits log --- Key: HDFS-988 URL: https://issues.apache.org/jira/browse/HDFS-988 Project: Hadoop HDFS Issue Type: Bug Components: name-node Reporter: dhruba borth

[jira] Commented: (HDFS-909) Race condition between rollEditLog or rollFSImage ant FSEditsLog.write operations corrupts edits log

[ https://issues.apache.org/jira/browse/HDFS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12835370#action_12835370 ] dhruba borthakur commented on HDFS-909: --- I recently experienced the race condition betw

[jira] Commented: (HDFS-909) Race condition between rollEditLog or rollFSImage ant FSEditsLog.write operations corrupts edits log

[ https://issues.apache.org/jira/browse/HDFS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12835169#action_12835169 ] dhruba borthakur commented on HDFS-909: --- One problem I can see in 0.20 is that isSyncRu

[jira] Commented: (HDFS-909) Race condition between rollEditLog or rollFSImage ant FSEditsLog.write operations corrupts edits log

[ https://issues.apache.org/jira/browse/HDFS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12835168#action_12835168 ] dhruba borthakur commented on HDFS-909: --- I think this problem does not exist in 0.20. B

[jira] Commented: (HDFS-984) Delegation Tokens should be persisted in Namenode

[ https://issues.apache.org/jira/browse/HDFS-984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12835128#action_12835128 ] dhruba borthakur commented on HDFS-984: --- Can you please explain the use-case why these

[jira] Commented: (HDFS-985) HDFS should issue multiple RPCs for listing a large directory

[ https://issues.apache.org/jira/browse/HDFS-985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12835127#action_12835127 ] dhruba borthakur commented on HDFS-985: --- +1. sounds like an awesome idea. > HDFS sho

[jira] Commented: (HDFS-915) Hung DN stalls write pipeline for far longer than its timeout

[ https://issues.apache.org/jira/browse/HDFS-915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12835102#action_12835102 ] dhruba borthakur commented on HDFS-915: --- My suggestion would be to set the timeout smal

[jira] Updated: (HDFS-826) Allow a mechanism for an application to detect that datanode(s) have died in the write pipeline

[ https://issues.apache.org/jira/browse/HDFS-826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-826: -- Attachment: ReplicableHdfs3.txt Marged patch with latest trunk. > Allow a mechanism for an appli

[jira] Updated: (HDFS-142) Datanode should delete files under tmp when upgraded from 0.17

[ https://issues.apache.org/jira/browse/HDFS-142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-142: -- Tags: hbase This patch is needed to make Hbase work correctly on Hadoop 0.20. > Datanode should

[jira] Updated: (HDFS-200) In HDFS, sync() not yet guarantees data available to the new readers

[ https://issues.apache.org/jira/browse/HDFS-200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-200: -- Tags: hbase This patch is needed to make Hbase work correctly on Hadoop 0.20. > In HDFS, sync()

[jira] Updated: (HDFS-826) Allow a mechanism for an application to detect that datanode(s) have died in the write pipeline

[ https://issues.apache.org/jira/browse/HDFS-826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-826: -- Tags: hbase This patch is needed to make Hbase work elegantly on Hadoop 0.20. > Allow a mechanis

[jira] Commented: (HDFS-915) Hung DN stalls write pipeline for far longer than its timeout

[ https://issues.apache.org/jira/browse/HDFS-915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12835061#action_12835061 ] dhruba borthakur commented on HDFS-915: --- when I first wrote some of this code in 0.16 o

[jira] Commented: (HDFS-915) Hung DN stalls write pipeline for far longer than its timeout

[ https://issues.apache.org/jira/browse/HDFS-915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12835048#action_12835048 ] dhruba borthakur commented on HDFS-915: --- I agree with Todd that the default value of Hd

[jira] Commented: (HDFS-983) NameNode transaction log corruption with bad filename

[ https://issues.apache.org/jira/browse/HDFS-983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12835038#action_12835038 ] dhruba borthakur commented on HDFS-983: --- Hi todd, thanks for the pointer to HADOOP-6522

[jira] Assigned: (HDFS-983) NameNode transaction log corruption with bad filename

[ https://issues.apache.org/jira/browse/HDFS-983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur reassigned HDFS-983: - Assignee: dhruba borthakur > NameNode transaction log corruption with bad filename > --

[jira] Commented: (HDFS-983) NameNode transaction log corruption with bad filename

[ https://issues.apache.org/jira/browse/HDFS-983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834994#action_12834994 ] dhruba borthakur commented on HDFS-983: --- {code} 2010-02-17 00:03:23,763 ERROR org.apac

[jira] Created: (HDFS-983) NameNode transaction log corruption with bad filename

NameNode transaction log corruption with bad filename - Key: HDFS-983 URL: https://issues.apache.org/jira/browse/HDFS-983 Project: Hadoop HDFS Issue Type: Bug Components: name-nod

[jira] Commented: (HDFS-729) fsck option to list only corrupted files

[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834107#action_12834107 ] dhruba borthakur commented on HDFS-729: --- One option would be to add ClientProtocol.getB

[jira] Commented: (HDFS-909) Race condition between rollEditLog or rollFSImage ant FSEditsLog.write operations corrupts edits log

[ https://issues.apache.org/jira/browse/HDFS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834077#action_12834077 ] dhruba borthakur commented on HDFS-909: --- I agree that the RWlock idea is only for trunk

[jira] Commented: (HDFS-909) Race condition between rollEditLog or rollFSImage ant FSEditsLog.write operations corrupts edits log

[ https://issues.apache.org/jira/browse/HDFS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834051#action_12834051 ] dhruba borthakur commented on HDFS-909: --- I agree with Todd's latest observation. i thin

[jira] Commented: (HDFS-978) Record every new block allocation of a file into the transaction log.

[ https://issues.apache.org/jira/browse/HDFS-978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833622#action_12833622 ] dhruba borthakur commented on HDFS-978: --- When a writer requests a new block for a file,

[jira] Updated: (HDFS-976) Hot Standby for NameNode

[ https://issues.apache.org/jira/browse/HDFS-976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-976: -- Attachment: AvatarNode.patch AvatarNodeDescription.txt A short writeup and code t

[jira] Created: (HDFS-978) Record every new block allocation of a file into the transaction log.

Record every new block allocation of a file into the transaction log. - Key: HDFS-978 URL: https://issues.apache.org/jira/browse/HDFS-978 Project: Hadoop HDFS Issue Type: Im

[jira] Created: (HDFS-976) Hot Standby for NameNode

Hot Standby for NameNode Key: HDFS-976 URL: https://issues.apache.org/jira/browse/HDFS-976 Project: Hadoop HDFS Issue Type: New Feature Components: name-node Reporter: dhruba borthakur

[jira] Updated: (HDFS-826) Allow a mechanism for an application to detect that datanode(s) have died in the write pipeline

[ https://issues.apache.org/jira/browse/HDFS-826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-826: -- Attachment: ReplicableHdfs2.txt I addressed Hairong's concern and kept this confined to DFSClient

[jira] Commented: (HDFS-970) FSImage writing should always fsync before close

[ https://issues.apache.org/jira/browse/HDFS-970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12832799#action_12832799 ] dhruba borthakur commented on HDFS-970: --- This is a good thing to do. This could increas

[jira] Commented: (HDFS-931) DFSClient.close doesn't rethrow exceptions thrown by automatically closed clients

[ https://issues.apache.org/jira/browse/HDFS-931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12832113#action_12832113 ] dhruba borthakur commented on HDFS-931: --- I think we should make this change only in tru

[jira] Commented: (HDFS-931) DFSClient.close doesn't rethrow exceptions thrown by automatically closed clients

[ https://issues.apache.org/jira/browse/HDFS-931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831739#action_12831739 ] dhruba borthakur commented on HDFS-931: --- This could break existing applications in a ba

[jira] Commented: (HDFS-931) DFSClient.close doesn't rethrow exceptions thrown by automatically closed clients

[ https://issues.apache.org/jira/browse/HDFS-931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831733#action_12831733 ] dhruba borthakur commented on HDFS-931: --- > Is there any value in letting exceptions out

[jira] Commented: (HDFS-894) DatanodeID.ipcPort is not updated when existing node re-registers

[ https://issues.apache.org/jira/browse/HDFS-894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831730#action_12831730 ] dhruba borthakur commented on HDFS-894: --- The code looks good. But since this is not a r

[jira] Created: (HDFS-966) NameNode recovers lease even in safemode

NameNode recovers lease even in safemode Key: HDFS-966 URL: https://issues.apache.org/jira/browse/HDFS-966 Project: Hadoop HDFS Issue Type: Bug Components: name-node Reporter: dh

[jira] Commented: (HDFS-957) FSImage layout version should be only once file is complete

[ https://issues.apache.org/jira/browse/HDFS-957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831615#action_12831615 ] dhruba borthakur commented on HDFS-957: --- I like your idea of using FileChannel.force().

[jira] Commented: (HDFS-957) FSImage layout version should be only once file is complete

[ https://issues.apache.org/jira/browse/HDFS-957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831580#action_12831580 ] dhruba borthakur commented on HDFS-957: --- Ok, I see your point. Suppose the NN issues t

[jira] Commented: (HDFS-959) Performance improvements to DFSClient and DataNode for faster DFS write at replication factor of 1

[ https://issues.apache.org/jira/browse/HDFS-959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831576#action_12831576 ] dhruba borthakur commented on HDFS-959: --- > We currently allow 80 packets (~ 5MB) worth

[jira] Commented: (HDFS-957) FSImage layout version should be only once file is complete

[ https://issues.apache.org/jira/browse/HDFS-957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831380#action_12831380 ] dhruba borthakur commented on HDFS-957: --- Hi Todd, Can the NN write a trailer record to

[jira] Commented: (HDFS-959) Performance improvements to DFSClient and DataNode for faster DFS write at replication factor of 1

[ https://issues.apache.org/jira/browse/HDFS-959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831375#action_12831375 ] dhruba borthakur commented on HDFS-959: --- This is good stuff. > Performance improvemen

[jira] Updated: (HDFS-200) In HDFS, sync() not yet guarantees data available to the new readers

[ https://issues.apache.org/jira/browse/HDFS-200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-200: -- Attachment: fsyncConcurrentReaders16_20.txt Incorporated Todd's review comments. Thanks Todd. The

[jira] Commented: (HDFS-200) In HDFS, sync() not yet guarantees data available to the new readers

[ https://issues.apache.org/jira/browse/HDFS-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12830584#action_12830584 ] dhruba borthakur commented on HDFS-200: --- Hi Todd, I think your proposal might not work.

[jira] Commented: (HDFS-200) In HDFS, sync() not yet guarantees data available to the new readers

[ https://issues.apache.org/jira/browse/HDFS-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12830417#action_12830417 ] dhruba borthakur commented on HDFS-200: --- I meant "For a HDFS file, only the last block

[jira] Commented: (HDFS-200) In HDFS, sync() not yet guarantees data available to the new readers

[ https://issues.apache.org/jira/browse/HDFS-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12830416#action_12830416 ] dhruba borthakur commented on HDFS-200: --- Hi Todd, I think your point of concern deems i

[jira] Updated: (HDFS-200) In HDFS, sync() not yet guarantees data available to the new readers

[ https://issues.apache.org/jira/browse/HDFS-200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-200: -- Attachment: fsyncConcurrentReaders15_20.txt Merged with latest trunk > In HDFS, sync() not yet g

[jira] Commented: (HDFS-951) DFSClient should handle all nodes in a pipeline failed.

[ https://issues.apache.org/jira/browse/HDFS-951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12830125#action_12830125 ] dhruba borthakur commented on HDFS-951: --- If all datanodes in the pipleline are dead, th

[jira] Commented: (HDFS-946) NameNode should not return full path name when lisitng a diretory or getting the status of a file

[ https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829564#action_12829564 ] dhruba borthakur commented on HDFS-946: --- Client's should continue to get the full path

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829364#action_12829364 ] dhruba borthakur commented on HDFS-918: --- > because we want to push as many parallel rea

[jira] Updated: (HDFS-947) The namenode should redirect a hftp request to read a file to the datanode that has the maximum number of local replicas

[ https://issues.apache.org/jira/browse/HDFS-947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-947: -- Attachment: hftpRedirection.patch This is a patch fro hadoop 0.20. The namenode redirects a hftp

[jira] Created: (HDFS-947) The namenode should redirect a hftp request to read a file to the datanode that has the maximum number of local replicas

The namenode should redirect a hftp request to read a file to the datanode that has the maximum number of local replicas Key: HDFS-947 URL: ht

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806647#action_12806647 ] dhruba borthakur commented on HDFS-918: --- The CPU time needed to dynamically create a th

[jira] Commented: (HDFS-729) fsck option to list only corrupted files

[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806646#action_12806646 ] dhruba borthakur commented on HDFS-729: --- It appears that I have consensus from Raghu, K

[jira] Commented: (HDFS-895) Allow hflush/sync to occur in parallel with new writes to the file

[ https://issues.apache.org/jira/browse/HDFS-895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806645#action_12806645 ] dhruba borthakur commented on HDFS-895: --- Hi Todd, I am not working on this one yet and

[jira] Commented: (HDFS-916) Rewrite DFSOutputStream to use a single thread with NIO

[ https://issues.apache.org/jira/browse/HDFS-916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806442#action_12806442 ] dhruba borthakur commented on HDFS-916: --- I agree with your view. let's make HDFS-914 a

[jira] Commented: (HDFS-729) fsck option to list only corrupted files

[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806441#action_12806441 ] dhruba borthakur commented on HDFS-729: --- > o basically the burden on removing duplicate

[jira] Commented: (HDFS-916) Rewrite DFSOutputStream to use a single thread with NIO

[ https://issues.apache.org/jira/browse/HDFS-916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806420#action_12806420 ] dhruba borthakur commented on HDFS-916: --- In principle, I am fine with a rewrite of DFSO

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806419#action_12806419 ] dhruba borthakur commented on HDFS-918: --- This is good work. > it could make things si

[jira] Commented: (HDFS-920) Incorrect metrics reporting of transcations metrics.

[ https://issues.apache.org/jira/browse/HDFS-920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806418#action_12806418 ] dhruba borthakur commented on HDFS-920: --- I am guessing that the metric currently tracks

[jira] Commented: (HDFS-929) DFSClient#getBlockSize is unused

[ https://issues.apache.org/jira/browse/HDFS-929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806411#action_12806411 ] dhruba borthakur commented on HDFS-929: --- Maybe you can mark it as deprecated? > DFSCli

[jira] Commented: (HDFS-729) fsck option to list only corrupted files

[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805520#action_12805520 ] dhruba borthakur commented on HDFS-729: --- Hi Hairong and Raghu, I had a discussion with

[jira] Assigned: (HDFS-854) Datanode should scan devices in parallel to generate block report

[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur reassigned HDFS-854: - Assignee: Dmytro Molkov > Datanode should scan devices in parallel to generate block report

<    2   3   4   5   6   7   8   9   10   11   >