[jira] [Updated] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[ https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11530: - Fix Version/s: 2.9.0 > Use HDFS specific network topology to choose datanode in > BlockPlacementPolicyDefault > > > Key: HDFS-11530 > URL: https://issues.apache.org/jira/browse/HDFS-11530 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0-alpha2 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Fix For: 2.9.0, 3.0.0-alpha4 > > Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch, > HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch, > HDFS-11530.006.patch, HDFS-11530.007.patch, HDFS-11530.008.patch, > HDFS-11530.009.patch, HDFS-11530.010.patch, HDFS-11530.011.patch, > HDFS-11530.012.patch, HDFS-11530.013.patch, HDFS-11530.014.patch, > HDFS-11530.015.patch > > > The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482. > But this method is contained in new topology {{DFSNetworkTopology}} which is > specified for HDFS. We should update this and let > {{BlockPlacementPolicyDefault}} use the new way since the original way is > inefficient. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[ https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11530: - Description: The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482. But this method is contained in new topology {{DFSNetworkTopology}} which is specified for HDFS. We should update this and let {{BlockPlacementPolicyDefault}} use the new way since the original way is inefficient. (was: The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482. But this method is contained in new topology {{DFSNetworkTopology}} which is specified for HDFS. We should update this and let {{BlockPlacementPolicyDefault}} use the new way.) > Use HDFS specific network topology to choose datanode in > BlockPlacementPolicyDefault > > > Key: HDFS-11530 > URL: https://issues.apache.org/jira/browse/HDFS-11530 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0-alpha2 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: HDFS-11530.001.patch > > > The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482. > But this method is contained in new topology {{DFSNetworkTopology}} which is > specified for HDFS. We should update this and let > {{BlockPlacementPolicyDefault}} use the new way since the original way is > inefficient. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[ https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11530: - Status: Patch Available (was: Open) > Use HDFS specific network topology to choose datanode in > BlockPlacementPolicyDefault > > > Key: HDFS-11530 > URL: https://issues.apache.org/jira/browse/HDFS-11530 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0-alpha2 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: HDFS-11530.001.patch > > > The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482. > But this method is contained in new topology {{DFSNetworkTopology}} which is > specified for HDFS. We should update this and let > {{BlockPlacementPolicyDefault}} use the new way since the original way is > inefficient. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[ https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11530: - Attachment: HDFS-11530.001.patch Initial patch attached. Kindly review. > Use HDFS specific network topology to choose datanode in > BlockPlacementPolicyDefault > > > Key: HDFS-11530 > URL: https://issues.apache.org/jira/browse/HDFS-11530 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0-alpha2 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: HDFS-11530.001.patch > > > The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482. > But this method is contained in new topology {{DFSNetworkTopology}} which is > specified for HDFS. We should update this and let > {{BlockPlacementPolicyDefault}} use the new way since the original way is > inefficient. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[ https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11530: - Attachment: HDFS-11530.002.patch Attach the new patch on this JIRA as HDFS-11577 has been merged to trunk. > Use HDFS specific network topology to choose datanode in > BlockPlacementPolicyDefault > > > Key: HDFS-11530 > URL: https://issues.apache.org/jira/browse/HDFS-11530 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0-alpha2 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch > > > The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482. > But this method is contained in new topology {{DFSNetworkTopology}} which is > specified for HDFS. We should update this and let > {{BlockPlacementPolicyDefault}} use the new way since the original way is > inefficient. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[ https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11530: -- Attachment: HDFS-11530.003.patch Hi [~linyiqun] I think I've identified one issue, which is that the code does not handle heartbeat. Specifically, it appears to me that it works in the way that a node is added first, but with an empty storage info at this point. Later when the first heartbeat comes, {{DatanodeDescriptor}} gets updated and the storage info gets filled. Namely, {{DFSTopologyNodeImpl}} needs to be updated accordingly on receiving heartbeats, this is the missing piece causing at least part of the problem. Also, adding the heart-triggered-updates might just handle datanode failover situation, but this needs to be verified. Post v003 patch here to add updates of storage types on heartbeat. The high level idea is that when a heartbeat indicates a datanode has storage type add/remove, inform the parent recursively to have all the counters updated. > Use HDFS specific network topology to choose datanode in > BlockPlacementPolicyDefault > > > Key: HDFS-11530 > URL: https://issues.apache.org/jira/browse/HDFS-11530 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0-alpha2 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch, > HDFS-11530.003.patch > > > The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482. > But this method is contained in new topology {{DFSNetworkTopology}} which is > specified for HDFS. We should update this and let > {{BlockPlacementPolicyDefault}} use the new way since the original way is > inefficient. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[ https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11530: -- Attachment: HDFS-11530.004.patch Thanks [~arpitagarwal] for pointing out that {{pruneStorageMap()}} should also be modified accordingly, post v004 patch to add this. > Use HDFS specific network topology to choose datanode in > BlockPlacementPolicyDefault > > > Key: HDFS-11530 > URL: https://issues.apache.org/jira/browse/HDFS-11530 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0-alpha2 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch, > HDFS-11530.003.patch, HDFS-11530.004.patch > > > The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482. > But this method is contained in new topology {{DFSNetworkTopology}} which is > specified for HDFS. We should update this and let > {{BlockPlacementPolicyDefault}} use the new way since the original way is > inefficient. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[ https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11530: - Attachment: HDFS-11530.005.patch Thanks for your analysis, [~vagarychen]! I think there is no need to use {{DFSNetworkTopology}} in {{Dispatcher}}. As I look into the code, the {{NetworkTopology}} that used in class {{Dispatcher}} is completely independent. It will add the nodes into topology in the {{.Dispatcher#init}} method. In addition, the method {{NetworkTopology#isNodeGroupAware()}} will also be invoked in {{Dispatcher}}. Attach the new patch to address this and fix checkstyle warning. > Use HDFS specific network topology to choose datanode in > BlockPlacementPolicyDefault > > > Key: HDFS-11530 > URL: https://issues.apache.org/jira/browse/HDFS-11530 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0-alpha2 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch, > HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch > > > The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482. > But this method is contained in new topology {{DFSNetworkTopology}} which is > specified for HDFS. We should update this and let > {{BlockPlacementPolicyDefault}} use the new way since the original way is > inefficient. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[ https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11530: - Attachment: HDFS-11530.006.patch It seems that some tests are still failed due to different {{BlockPlacement}} plociy used in HDFS. I am thinking one reasonable way that we can use {{DFSNetworkTopology}} only if the current block placement is the default way(BlockPlacementPolicyRackFaultTolerant). Otherwise, we use the {{NetworkTopology#getInstance(conf)}}. Attach the updated patch. Any comments? [~vagarychen], [~arpitagarwal]. > Use HDFS specific network topology to choose datanode in > BlockPlacementPolicyDefault > > > Key: HDFS-11530 > URL: https://issues.apache.org/jira/browse/HDFS-11530 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0-alpha2 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch, > HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch, > HDFS-11530.006.patch > > > The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482. > But this method is contained in new topology {{DFSNetworkTopology}} which is > specified for HDFS. We should update this and let > {{BlockPlacementPolicyDefault}} use the new way since the original way is > inefficient. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[ https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11530: - Attachment: HDFS-11530.007.patch Post the new patch to fix findbug warning and update default policy used in {{DatanodeManager}}). > Use HDFS specific network topology to choose datanode in > BlockPlacementPolicyDefault > > > Key: HDFS-11530 > URL: https://issues.apache.org/jira/browse/HDFS-11530 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0-alpha2 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch, > HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch, > HDFS-11530.006.patch, HDFS-11530.007.patch > > > The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482. > But this method is contained in new topology {{DFSNetworkTopology}} which is > specified for HDFS. We should update this and let > {{BlockPlacementPolicyDefault}} use the new way since the original way is > inefficient. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[ https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11530: -- Attachment: HDFS-11530.008.patch Attaching v008 patch to fix the issue. [~linyiqun] [~arpitagarwal] would you please take a look on this one some time? As v008 patch made a few small changes but they kind of break some earlier assumptions. More specifically, {{TestBlockManager.testSafeModeIBR}} fail seems to be caused by that the {{add(Node node)}} was called more than once on the same node, with different storage type info. This violates the earlier assumption that nodes only get called by add once, or at least always the same storage info. The reason seems that this tests is simulating datanode fails and this is why nodes get added twice. This seems to be a valid case so I added {{DFSTopologyNodeImpl.updateExistingDatanode}} for this. On the other hand, {{TestBlockStatsMXBean.testStorageTypeStatsWhenStorageFailed}} fail seems to be caused by that the {{excludedNodes}} set does not have to be {{DatanodeDescriptor}}, but can also be {{DatanodeInfo}} instead. (Although it's a bit suspicious to me why this is the only test that failed due to this). Assuming this is valid, I modified {{DFSNetworkTopology.chooseRandomWithStorageType}} to do some more check. I do appreciate any comments. Basically I made the changes here assuming the tests are doing the right thing, but it might very well be that the tests themselves should be modified instead. > Use HDFS specific network topology to choose datanode in > BlockPlacementPolicyDefault > > > Key: HDFS-11530 > URL: https://issues.apache.org/jira/browse/HDFS-11530 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0-alpha2 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch, > HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch, > HDFS-11530.006.patch, HDFS-11530.007.patch, HDFS-11530.008.patch > > > The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482. > But this method is contained in new topology {{DFSNetworkTopology}} which is > specified for HDFS. We should update this and let > {{BlockPlacementPolicyDefault}} use the new way since the original way is > inefficient. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[ https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11530: -- Attachment: HDFS-11530.009.patch Realized that there is a minor small bug in v008 patch, post v009 patch to address it. And thanks [~linyiqun] for sharing your thought! I do agree that we should definitely revisit tests to see whether they should be fixed based on what we found here. > Use HDFS specific network topology to choose datanode in > BlockPlacementPolicyDefault > > > Key: HDFS-11530 > URL: https://issues.apache.org/jira/browse/HDFS-11530 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0-alpha2 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch, > HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch, > HDFS-11530.006.patch, HDFS-11530.007.patch, HDFS-11530.008.patch, > HDFS-11530.009.patch > > > The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482. > But this method is contained in new topology {{DFSNetworkTopology}} which is > specified for HDFS. We should update this and let > {{BlockPlacementPolicyDefault}} use the new way since the original way is > inefficient. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[ https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11530: - Attachment: HDFS-11530.010.patch > Use HDFS specific network topology to choose datanode in > BlockPlacementPolicyDefault > > > Key: HDFS-11530 > URL: https://issues.apache.org/jira/browse/HDFS-11530 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0-alpha2 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch, > HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch, > HDFS-11530.006.patch, HDFS-11530.007.patch, HDFS-11530.008.patch, > HDFS-11530.009.patch, HDFS-11530.010.patch > > > The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482. > But this method is contained in new topology {{DFSNetworkTopology}} which is > specified for HDFS. We should update this and let > {{BlockPlacementPolicyDefault}} use the new way since the original way is > inefficient. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[ https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11530: - Attachment: HDFS-11530.011.patch > Use HDFS specific network topology to choose datanode in > BlockPlacementPolicyDefault > > > Key: HDFS-11530 > URL: https://issues.apache.org/jira/browse/HDFS-11530 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0-alpha2 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch, > HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch, > HDFS-11530.006.patch, HDFS-11530.007.patch, HDFS-11530.008.patch, > HDFS-11530.009.patch, HDFS-11530.010.patch, HDFS-11530.011.patch > > > The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482. > But this method is contained in new topology {{DFSNetworkTopology}} which is > specified for HDFS. We should update this and let > {{BlockPlacementPolicyDefault}} use the new way since the original way is > inefficient. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[ https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11530: - Attachment: HDFS-11530.012.patch Thanks [~arpitagarwal] for the review! {quote} If we fail to place a block for the first storage type then the function is guaranteed to fail later, correct? {quote} I don't think so on this. The first storage type failed doesn't mean the next differential type will failed too. We have a set of required storage type, if any one of storage type that node contained, then this node should be a good node. The related codes: {code} for (Iterator> iter = storageTypes .entrySet().iterator(); iter.hasNext();) { Map.Entry entry = iter.next(); storage = chooseStorage4Block( chosenNode, blocksize, results, entry.getKey()); } {code} The second comment makes sense to me. I have addressed the comment in my latest patch. BTW, I have refactor the new setting into core-default.xml. {quote} DatanodeDescriptor.java: The reflection checks at lines 503 and 901 also make me sad. {quote} I'm a little confused on this. Did you mean the following check is not a good solution? {code} if (getParent() instanceof DFSTopologyNodeImpl) {code} Attach the new patch to mainly address the comment#2. Thanks for the review. > Use HDFS specific network topology to choose datanode in > BlockPlacementPolicyDefault > > > Key: HDFS-11530 > URL: https://issues.apache.org/jira/browse/HDFS-11530 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0-alpha2 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch, > HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch, > HDFS-11530.006.patch, HDFS-11530.007.patch, HDFS-11530.008.patch, > HDFS-11530.009.patch, HDFS-11530.010.patch, HDFS-11530.011.patch, > HDFS-11530.012.patch > > > The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482. > But this method is contained in new topology {{DFSNetworkTopology}} which is > specified for HDFS. We should update this and let > {{BlockPlacementPolicyDefault}} use the new way since the original way is > inefficient. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[ https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11530: - Attachment: HDFS-11530.012.patch > Use HDFS specific network topology to choose datanode in > BlockPlacementPolicyDefault > > > Key: HDFS-11530 > URL: https://issues.apache.org/jira/browse/HDFS-11530 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0-alpha2 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch, > HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch, > HDFS-11530.006.patch, HDFS-11530.007.patch, HDFS-11530.008.patch, > HDFS-11530.009.patch, HDFS-11530.010.patch, HDFS-11530.011.patch, > HDFS-11530.012.patch > > > The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482. > But this method is contained in new topology {{DFSNetworkTopology}} which is > specified for HDFS. We should update this and let > {{BlockPlacementPolicyDefault}} use the new way since the original way is > inefficient. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[ https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11530: - Attachment: (was: HDFS-11530.012.patch) > Use HDFS specific network topology to choose datanode in > BlockPlacementPolicyDefault > > > Key: HDFS-11530 > URL: https://issues.apache.org/jira/browse/HDFS-11530 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0-alpha2 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch, > HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch, > HDFS-11530.006.patch, HDFS-11530.007.patch, HDFS-11530.008.patch, > HDFS-11530.009.patch, HDFS-11530.010.patch, HDFS-11530.011.patch, > HDFS-11530.012.patch > > > The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482. > But this method is contained in new topology {{DFSNetworkTopology}} which is > specified for HDFS. We should update this and let > {{BlockPlacementPolicyDefault}} use the new way since the original way is > inefficient. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[ https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11530: - Attachment: HDFS-11530.013.patch Thanks [~vagarychen] for providing the concrete analysis! In my last comment, I am focus on the point in the follwoing codes: {code} for (StorageType type : storageTypes.keySet()) { ... } {code} Seemed I was misreading the comment from [~arpitagarwal], :). Almost agree on your comment, [~vagarychen]. It's a nice catch. Actually in the new logic, we still need to do some same logic(contained in loop). So I'd like to make a minor adjustment that just skip the other storage type in loop and then I can reuse the most of existing codes. Attach the new patch. Please have a review. > Use HDFS specific network topology to choose datanode in > BlockPlacementPolicyDefault > > > Key: HDFS-11530 > URL: https://issues.apache.org/jira/browse/HDFS-11530 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0-alpha2 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch, > HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch, > HDFS-11530.006.patch, HDFS-11530.007.patch, HDFS-11530.008.patch, > HDFS-11530.009.patch, HDFS-11530.010.patch, HDFS-11530.011.patch, > HDFS-11530.012.patch, HDFS-11530.013.patch > > > The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482. > But this method is contained in new topology {{DFSNetworkTopology}} which is > specified for HDFS. We should update this and let > {{BlockPlacementPolicyDefault}} use the new way since the original way is > inefficient. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[ https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11530: - Attachment: HDFS-11530.014.patch Thanks [~arpitagarwal] for the review! Post the new patch to address the comment. > Use HDFS specific network topology to choose datanode in > BlockPlacementPolicyDefault > > > Key: HDFS-11530 > URL: https://issues.apache.org/jira/browse/HDFS-11530 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0-alpha2 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch, > HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch, > HDFS-11530.006.patch, HDFS-11530.007.patch, HDFS-11530.008.patch, > HDFS-11530.009.patch, HDFS-11530.010.patch, HDFS-11530.011.patch, > HDFS-11530.012.patch, HDFS-11530.013.patch, HDFS-11530.014.patch > > > The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482. > But this method is contained in new topology {{DFSNetworkTopology}} which is > specified for HDFS. We should update this and let > {{BlockPlacementPolicyDefault}} use the new way since the original way is > inefficient. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[ https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11530: - Attachment: HDFS-11530.015.patch The test failures are not related. Post the new patch to fix checkstyle issues. > Use HDFS specific network topology to choose datanode in > BlockPlacementPolicyDefault > > > Key: HDFS-11530 > URL: https://issues.apache.org/jira/browse/HDFS-11530 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0-alpha2 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch, > HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch, > HDFS-11530.006.patch, HDFS-11530.007.patch, HDFS-11530.008.patch, > HDFS-11530.009.patch, HDFS-11530.010.patch, HDFS-11530.011.patch, > HDFS-11530.012.patch, HDFS-11530.013.patch, HDFS-11530.014.patch, > HDFS-11530.015.patch > > > The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482. > But this method is contained in new topology {{DFSNetworkTopology}} which is > specified for HDFS. We should update this and let > {{BlockPlacementPolicyDefault}} use the new way since the original way is > inefficient. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[ https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11530: - Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha3 > Use HDFS specific network topology to choose datanode in > BlockPlacementPolicyDefault > > > Key: HDFS-11530 > URL: https://issues.apache.org/jira/browse/HDFS-11530 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0-alpha2 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Fix For: 3.0.0-alpha3 > > Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch, > HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch, > HDFS-11530.006.patch, HDFS-11530.007.patch, HDFS-11530.008.patch, > HDFS-11530.009.patch, HDFS-11530.010.patch, HDFS-11530.011.patch, > HDFS-11530.012.patch, HDFS-11530.013.patch, HDFS-11530.014.patch, > HDFS-11530.015.patch > > > The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482. > But this method is contained in new topology {{DFSNetworkTopology}} which is > specified for HDFS. We should update this and let > {{BlockPlacementPolicyDefault}} use the new way since the original way is > inefficient. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[ https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11530: - Resolution: Fixed Status: Resolved (was: Patch Available) > Use HDFS specific network topology to choose datanode in > BlockPlacementPolicyDefault > > > Key: HDFS-11530 > URL: https://issues.apache.org/jira/browse/HDFS-11530 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0-alpha2 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Fix For: 3.0.0-alpha3 > > Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch, > HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch, > HDFS-11530.006.patch, HDFS-11530.007.patch, HDFS-11530.008.patch, > HDFS-11530.009.patch, HDFS-11530.010.patch, HDFS-11530.011.patch, > HDFS-11530.012.patch, HDFS-11530.013.patch, HDFS-11530.014.patch, > HDFS-11530.015.patch > > > The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482. > But this method is contained in new topology {{DFSNetworkTopology}} which is > specified for HDFS. We should update this and let > {{BlockPlacementPolicyDefault}} use the new way since the original way is > inefficient. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org