[jira] [Commented] (HDFS-13639) SlotReleaser is not fast enough

2018-07-28 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560942#comment-16560942 ] Gang Xie commented on HDFS-13639: - Attached the diff against the 2.4. > SlotReleaser is not fast enough

[jira] [Updated] (HDFS-13639) SlotReleaser is not fast enough

2018-07-28 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Xie updated HDFS-13639: Attachment: HDFS-13639-2.4.diff > SlotReleaser is not fast enough > --- > >

[jira] [Issue Comment Deleted] (HDFS-13571) Dead datanode detector

2018-07-28 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Xie updated HDFS-13571: Comment: was deleted (was: shttps://issues.apache.org/jira/browse/HDFS-13571?filter=-2#sdfs) > Dead

[jira] [Commented] (HDFS-13571) Dead datanode detector

2018-07-28 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560941#comment-16560941 ] Gang Xie commented on HDFS-13571: - Attached the diff against 2.6 in case some others need it > Dead

[jira] [Commented] (HDFS-13571) Dead datanode detector

2018-07-28 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560939#comment-16560939 ] Gang Xie commented on HDFS-13571: - shttps://issues.apache.org/jira/browse/HDFS-13571?filter=-2#sdfs >

[jira] [Updated] (HDFS-13571) Dead datanode detector

2018-07-28 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Xie updated HDFS-13571: Attachment: HDFS-13571-2.6.diff > Dead datanode detector > -- > > Key:

[jira] [Commented] (HDFS-8865) Improve quota initialization performance

2018-06-01 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497816#comment-16497816 ] Gang Xie commented on HDFS-8865: We want to port this path to our 2.4 release. But we have a little

[jira] [Commented] (HDFS-13639) SlotReleaser is not fast enough

2018-05-30 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495014#comment-16495014 ] Gang Xie commented on HDFS-13639: - I used YCSB on hbase to issue local read on HDFDS datanode. Pls see

[jira] [Updated] (HDFS-13640) enable ShortCircuit Read on UC block

2018-05-30 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Xie updated HDFS-13640: Affects Version/s: 3.0.2 Description: The ShortCircuit read is disabled by HDFS-2757 due to the

[jira] [Created] (HDFS-13640) s

2018-05-30 Thread Gang Xie (JIRA)
Gang Xie created HDFS-13640: --- Summary: s Key: HDFS-13640 URL: https://issues.apache.org/jira/browse/HDFS-13640 Project: Hadoop HDFS Issue Type: Improvement Reporter: Gang Xie --

[jira] [Updated] (HDFS-13639) SlotReleaser is not fast enough

2018-05-30 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Xie updated HDFS-13639: Environment: 1. YCSB: {color:#00} recordcount=20 fieldcount=1 fieldlength=1000

[jira] [Created] (HDFS-13639) SlotReleaser is not fast enough

2018-05-30 Thread Gang Xie (JIRA)
Gang Xie created HDFS-13639: --- Summary: SlotReleaser is not fast enough Key: HDFS-13639 URL: https://issues.apache.org/jira/browse/HDFS-13639 Project: Hadoop HDFS Issue Type: Improvement

[jira] [Commented] (HDFS-13121) NPE when request file descriptors when SC read

2018-05-27 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492211#comment-16492211 ] Gang Xie commented on HDFS-13121: - Agree with that. It's hard to test such case with UT. so I didn't

[jira] [Created] (HDFS-13594) the lock of ShortCircuitCache is hold while close the ShortCircuitReplica

2018-05-18 Thread Gang Xie (JIRA)
Gang Xie created HDFS-13594: --- Summary: the lock of ShortCircuitCache is hold while close the ShortCircuitReplica Key: HDFS-13594 URL: https://issues.apache.org/jira/browse/HDFS-13594 Project: Hadoop HDFS

[jira] [Created] (HDFS-13571) Dead datanode detector

2018-05-15 Thread Gang Xie (JIRA)
Gang Xie created HDFS-13571: --- Summary: Dead datanode detector Key: HDFS-13571 URL: https://issues.apache.org/jira/browse/HDFS-13571 Project: Hadoop HDFS Issue Type: Improvement

[jira] [Created] (HDFS-13564) PreAllocator for DfsClientShm

2018-05-15 Thread Gang Xie (JIRA)
Gang Xie created HDFS-13564: --- Summary: PreAllocator for DfsClientShm Key: HDFS-13564 URL: https://issues.apache.org/jira/browse/HDFS-13564 Project: Hadoop HDFS Issue Type: Improvement

[jira] [Commented] (HDFS-13378) Add the option to set the du reserve by percent on datanode

2018-03-30 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420350#comment-16420350 ] Gang Xie commented on HDFS-13378: - attach the fix. > Add the option to set the du reserve by percent on

[jira] [Updated] (HDFS-13378) Add the option to set the du reserve by percent on datanode

2018-03-30 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Xie updated HDFS-13378: Attachment: HDFS-13378_v1.diff > Add the option to set the du reserve by percent on datanode >

[jira] [Created] (HDFS-13378) Add the option to set the du reserve by percent on datanode

2018-03-30 Thread Gang Xie (JIRA)
Gang Xie created HDFS-13378: --- Summary: Add the option to set the du reserve by percent on datanode Key: HDFS-13378 URL: https://issues.apache.org/jira/browse/HDFS-13378 Project: Hadoop HDFS Issue

[jira] [Created] (HDFS-13121) NPE when request file descriptors when SC read

2018-02-07 Thread Gang Xie (JIRA)
Gang Xie created HDFS-13121: --- Summary: NPE when request file descriptors when SC read Key: HDFS-13121 URL: https://issues.apache.org/jira/browse/HDFS-13121 Project: Hadoop HDFS Issue Type: Bug

[jira] [Commented] (HDFS-6440) Support more than 2 NameNodes

2017-12-11 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16285652#comment-16285652 ] Gang Xie commented on HDFS-6440: Do we initiate the back-port of this feature to branch-2? According to my

[jira] [Commented] (HDFS-12528) Short-circuit reads unnecessarily disabled for a long time

2017-11-29 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272240#comment-16272240 ] Gang Xie commented on HDFS-12528: - What's the risk if we don't disable the SCR at all when we get any

[jira] [Commented] (HDFS-12528) Short-circuit reads unnecessarily disabled for a long time

2017-11-28 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270285#comment-16270285 ] Gang Xie commented on HDFS-12528: - create a patch for case when block is invalid by balancer, which also

[jira] [Updated] (HDFS-12821) Block invalid IOException causes the DFSClient domain socket being disabled

2017-11-28 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-12821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Xie updated HDFS-12821: Attachment: HDFS-12821.patch When we receive the block invalid exception, this means it's moved to some

[jira] [Commented] (HDFS-12820) Decommissioned datanode is counted in service cause datanode allcating failure

2017-11-20 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-12820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258981#comment-16258981 ] Gang Xie commented on HDFS-12820: - Why we don't substract nodesInService when we complete the decommission

[jira] [Comment Edited] (HDFS-12820) Decommissioned datanode is counted in service cause datanode allcating failure

2017-11-20 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-12820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258974#comment-16258974 ] Gang Xie edited comment on HDFS-12820 at 11/20/17 9:11 AM: --- After carefully

[jira] [Reopened] (HDFS-12820) Decommissioned datanode is counted in service cause datanode allcating failure

2017-11-20 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-12820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Xie reopened HDFS-12820: - After carefully check the issue reported in HDFS-9279, and found this issue is not a its dup. this case is

[jira] [Commented] (HDFS-12820) Decommissioned datanode is counted in service cause datanode allcating failure

2017-11-20 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-12820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258975#comment-16258975 ] Gang Xie commented on HDFS-12820: - And I believe this issue still in the latest version > Decommissioned

[jira] [Commented] (HDFS-12820) Decommissioned datanode is counted in service cause datanode allcating failure

2017-11-19 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-12820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258834#comment-16258834 ] Gang Xie commented on HDFS-12820: - nice, let me check it out. > Decommissioned datanode is counted in

[jira] [Commented] (HDFS-12821) Block invalid IOException causes the DFSClient domain socket being disabled

2017-11-15 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-12821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254712#comment-16254712 ] Gang Xie commented on HDFS-12821: - yes, should be the same issue. > Block invalid IOException causes the

[jira] [Commented] (HDFS-12528) Short-circuit reads getting disabled frequently in certain scenarios

2017-11-15 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254710#comment-16254710 ] Gang Xie commented on HDFS-12528: - We hit the similar issue and created a jira : HDFS-12821 According to

[jira] [Created] (HDFS-12821) Block invalid IOException causes the DFSClient domain socket being disabled

2017-11-15 Thread Gang Xie (JIRA)
Gang Xie created HDFS-12821: --- Summary: Block invalid IOException causes the DFSClient domain socket being disabled Key: HDFS-12821 URL: https://issues.apache.org/jira/browse/HDFS-12821 Project: Hadoop HDFS

[jira] [Created] (HDFS-12820) Decommissioned datanode is counted in service cause datanode allcating failure

2017-11-15 Thread Gang Xie (JIRA)
Gang Xie created HDFS-12820: --- Summary: Decommissioned datanode is counted in service cause datanode allcating failure Key: HDFS-12820 URL: https://issues.apache.org/jira/browse/HDFS-12820 Project: Hadoop

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2017-01-03 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796824#comment-15796824 ] Gang Xie commented on HDFS-7784: About the GC activities, the following is the gstat output. And it do

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2017-01-03 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796809#comment-15796809 ] Gang Xie commented on HDFS-7784: The hardware info: CPU: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz with 24

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2017-01-03 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796797#comment-15796797 ] Gang Xie commented on HDFS-7784: The JVM setting: -Xmx102400m -Xms102400m -Xmn5508m

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2016-12-28 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784660#comment-15784660 ] Gang Xie commented on HDFS-7784: After make the AclStorage synchronized, it could works. the loading time

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2016-12-28 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784285#comment-15784285 ] Gang Xie commented on HDFS-7784: Found a potential issue while trying to back port this patch to 2.4. Pls

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2016-12-26 Thread Gang Xie (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15777928#comment-15777928 ] Gang Xie commented on HDFS-7784: Hello, Any update about this improvement? Loading huge image really takes