[jira] [Created] (HADOOP-17558) DistCp: Reduce memory usage using a fixed size ThreadPoolExecutor

2021-03-01 Thread Ayush Saxena (Jira)
Ayush Saxena created HADOOP-17558: - Summary: DistCp: Reduce memory usage using a fixed size ThreadPoolExecutor Key: HADOOP-17558 URL: https://issues.apache.org/jira/browse/HADOOP-17558 Project:

[jira] [Work logged] (HADOOP-17531) DistCp: Reduce memory usage on copying huge directories

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17531?focusedWorklogId=559843=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559843 ] ASF GitHub Bot logged work on HADOOP-17531: --- Author: ASF GitHub Bot

[GitHub] [hadoop] jojochuang commented on a change in pull request #2732: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories.

2021-03-01 Thread GitBox
jojochuang commented on a change in pull request #2732: URL: https://github.com/apache/hadoop/pull/2732#discussion_r585324372 ## File path: hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm ## @@ -362,6 +362,7 @@ Command Line Options | `-copybuffersize ` | Size of the

[jira] [Work logged] (HADOOP-17531) DistCp: Reduce memory usage on copying huge directories

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17531?focusedWorklogId=559842=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559842 ] ASF GitHub Bot logged work on HADOOP-17531: --- Author: ASF GitHub Bot

[GitHub] [hadoop] jojochuang commented on a change in pull request #2732: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories.

2021-03-01 Thread GitBox
jojochuang commented on a change in pull request #2732: URL: https://github.com/apache/hadoop/pull/2732#discussion_r585324372 ## File path: hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm ## @@ -362,6 +362,7 @@ Command Line Options | `-copybuffersize ` | Size of the

[jira] [Updated] (HADOOP-17531) DistCp: Reduce memory usage on copying huge directories

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-17531: Labels: pull-request-available (was: ) > DistCp: Reduce memory usage on copying huge

[jira] [Work logged] (HADOOP-17531) DistCp: Reduce memory usage on copying huge directories

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17531?focusedWorklogId=559838=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559838 ] ASF GitHub Bot logged work on HADOOP-17531: --- Author: ASF GitHub Bot

[GitHub] [hadoop] ayushtkn opened a new pull request #2732: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories.

2021-03-01 Thread GitBox
ayushtkn opened a new pull request #2732: URL: https://github.com/apache/hadoop/pull/2732 https://issues.apache.org/jira/browse/HADOOP-17531 This is an automated message from the Apache Git Service. To respond to the

[jira] [Updated] (HADOOP-17548) ABFS: Config for Mkdir overwrite

2021-03-01 Thread Sumangala Patki (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumangala Patki updated HADOOP-17548: - Description: The call to mkdirs with overwrite set to true results in an additional

[jira] [Work logged] (HADOOP-17552) The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistaken usage of the rpcTimeout configuratio

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17552?focusedWorklogId=559823=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559823 ] ASF GitHub Bot logged work on HADOOP-17552: --- Author: ASF GitHub Bot

[GitHub] [hadoop] functioner commented on pull request #2727: HADOOP-17552. The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mista

2021-03-01 Thread GitBox
functioner commented on pull request #2727: URL: https://github.com/apache/hadoop/pull/2727#issuecomment-788654337 @iwasakims I've updated the description of `ipc.client.rpc-timeout.ms` in hadoop/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml You can

[GitHub] [hadoop] bibinchundatt commented on pull request #2719: YARN-10649. fix RMNodeImpl.updateExistContainers leak

2021-03-01 Thread GitBox
bibinchundatt commented on pull request #2719: URL: https://github.com/apache/hadoop/pull/2719#issuecomment-788629039 @Neilxzn will take a look by this weekend This is an automated message from the Apache Git Service. To

[jira] [Work logged] (HADOOP-17552) The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistaken usage of the rpcTimeout configuratio

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17552?focusedWorklogId=559808=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559808 ] ASF GitHub Bot logged work on HADOOP-17552: --- Author: ASF GitHub Bot

[GitHub] [hadoop] functioner commented on pull request #2727: HADOOP-17552. The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mista

2021-03-01 Thread GitBox
functioner commented on pull request #2727: URL: https://github.com/apache/hadoop/pull/2727#issuecomment-788624136 > @functioner Just update the value of `CommonConfigurationKeys.IPC_CLIENT_RPC_TIMEOUT_DEFAULT` and core-default.xml if you want to change the default timeout.

[GitHub] [hadoop] Hexiaoqiao commented on pull request #2721: HDFS-15856: Make recover the pipeline in same packet exceed times for…

2021-03-01 Thread GitBox
Hexiaoqiao commented on pull request #2721: URL: https://github.com/apache/hadoop/pull/2721#issuecomment-788595297 Committed to trunk. Thanks @qizhu-lucas for your contributions. Thanks @ayushtkn and @jojochuang for your reviews.

[GitHub] [hadoop] Hexiaoqiao merged pull request #2721: HDFS-15856: Make recover the pipeline in same packet exceed times for…

2021-03-01 Thread GitBox
Hexiaoqiao merged pull request #2721: URL: https://github.com/apache/hadoop/pull/2721 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[jira] [Work logged] (HADOOP-17552) The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistaken usage of the rpcTimeout configuratio

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17552?focusedWorklogId=559758=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559758 ] ASF GitHub Bot logged work on HADOOP-17552: --- Author: ASF GitHub Bot

[GitHub] [hadoop] iwasakims commented on pull request #2727: HADOOP-17552. The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistak

2021-03-01 Thread GitBox
iwasakims commented on pull request #2727: URL: https://github.com/apache/hadoop/pull/2727#issuecomment-788576411 @functioner Just update the value of `CommonConfigurationKeys.IPC_CLIENT_RPC_TIMEOUT_DEFAULT` and core-default.xml if you want to change the default timeout.

[GitHub] [hadoop] Neilxzn commented on pull request #2719: YARN-10649. fix RMNodeImpl.updateExistContainers leak

2021-03-01 Thread GitBox
Neilxzn commented on pull request #2719: URL: https://github.com/apache/hadoop/pull/2719#issuecomment-788551894 @bibinchundatt Can you please take a look at this PR? thx This is an automated message from the Apache Git

[jira] [Work logged] (HADOOP-17552) The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistaken usage of the rpcTimeout configuratio

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17552?focusedWorklogId=559722=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559722 ] ASF GitHub Bot logged work on HADOOP-17552: --- Author: ASF GitHub Bot

[GitHub] [hadoop] functioner removed a comment on pull request #2727: HADOOP-17552. The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to t

2021-03-01 Thread GitBox
functioner removed a comment on pull request #2727: URL: https://github.com/apache/hadoop/pull/2727#issuecomment-788507261 > > According to the comment in that test case, it "should not time out because effective rpc-timeout is multiple of ping interval: 1600 (= 800 * (1000 / 800 + 1))",

[jira] [Work logged] (HADOOP-17552) The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistaken usage of the rpcTimeout configuratio

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17552?focusedWorklogId=559721=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559721 ] ASF GitHub Bot logged work on HADOOP-17552: --- Author: ASF GitHub Bot

[GitHub] [hadoop] functioner removed a comment on pull request #2727: HADOOP-17552. The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to t

2021-03-01 Thread GitBox
functioner removed a comment on pull request #2727: URL: https://github.com/apache/hadoop/pull/2727#issuecomment-788419641 > > According to the comment in that test case, it "should not time out because effective rpc-timeout is multiple of ping interval: 1600 (= 800 * (1000 / 800 + 1))",

[jira] [Work logged] (HADOOP-17552) The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistaken usage of the rpcTimeout configuratio

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17552?focusedWorklogId=559712=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559712 ] ASF GitHub Bot logged work on HADOOP-17552: --- Author: ASF GitHub Bot

[jira] [Work logged] (HADOOP-17552) The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistaken usage of the rpcTimeout configuratio

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17552?focusedWorklogId=559713=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559713 ] ASF GitHub Bot logged work on HADOOP-17552: --- Author: ASF GitHub Bot

[GitHub] [hadoop] functioner edited a comment on pull request #2727: HADOOP-17552. The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to th

2021-03-01 Thread GitBox
functioner edited a comment on pull request #2727: URL: https://github.com/apache/hadoop/pull/2727#issuecomment-788518502 > > According to the comment in that test case, it "should not time out because effective rpc-timeout is multiple of ping interval: 1600 (= 800 * (1000 / 800 + 1))",

[GitHub] [hadoop] functioner edited a comment on pull request #2727: HADOOP-17552. The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to th

2021-03-01 Thread GitBox
functioner edited a comment on pull request #2727: URL: https://github.com/apache/hadoop/pull/2727#issuecomment-788518502 > > According to the comment in that test case, it "should not time out because effective rpc-timeout is multiple of ping interval: 1600 (= 800 * (1000 / 800 + 1))",

[jira] [Work logged] (HADOOP-17552) The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistaken usage of the rpcTimeout configuratio

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17552?focusedWorklogId=559711=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559711 ] ASF GitHub Bot logged work on HADOOP-17552: --- Author: ASF GitHub Bot

[GitHub] [hadoop] functioner edited a comment on pull request #2727: HADOOP-17552. The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to th

2021-03-01 Thread GitBox
functioner edited a comment on pull request #2727: URL: https://github.com/apache/hadoop/pull/2727#issuecomment-788518502 > > According to the comment in that test case, it "should not time out because effective rpc-timeout is multiple of ping interval: 1600 (= 800 * (1000 / 800 + 1))",

[jira] [Work logged] (HADOOP-17552) The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistaken usage of the rpcTimeout configuratio

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17552?focusedWorklogId=559710=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559710 ] ASF GitHub Bot logged work on HADOOP-17552: --- Author: ASF GitHub Bot

[GitHub] [hadoop] functioner edited a comment on pull request #2727: HADOOP-17552. The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to th

2021-03-01 Thread GitBox
functioner edited a comment on pull request #2727: URL: https://github.com/apache/hadoop/pull/2727#issuecomment-788518502 > > According to the comment in that test case, it "should not time out because effective rpc-timeout is multiple of ping interval: 1600 (= 800 * (1000 / 800 + 1))",

[GitHub] [hadoop] qizhu-lucas commented on pull request #2721: HDFS-15856: Make recover the pipeline in same packet exceed times for…

2021-03-01 Thread GitBox
qizhu-lucas commented on pull request #2721: URL: https://github.com/apache/hadoop/pull/2721#issuecomment-788521571 @jojochuang Could you help check the latest patch? Thanks. This is an automated message from the

[jira] [Work logged] (HADOOP-17552) The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistaken usage of the rpcTimeout configuratio

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17552?focusedWorklogId=559703=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559703 ] ASF GitHub Bot logged work on HADOOP-17552: --- Author: ASF GitHub Bot

[jira] [Work logged] (HADOOP-17552) The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistaken usage of the rpcTimeout configuratio

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17552?focusedWorklogId=559702=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559702 ] ASF GitHub Bot logged work on HADOOP-17552: --- Author: ASF GitHub Bot

[GitHub] [hadoop] functioner edited a comment on pull request #2727: HADOOP-17552. The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to th

2021-03-01 Thread GitBox
functioner edited a comment on pull request #2727: URL: https://github.com/apache/hadoop/pull/2727#issuecomment-788518502 > > According to the comment in that test case, it "should not time out because effective rpc-timeout is multiple of ping interval: 1600 (= 800 * (1000 / 800 + 1))",

[GitHub] [hadoop] functioner edited a comment on pull request #2727: HADOOP-17552. The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to th

2021-03-01 Thread GitBox
functioner edited a comment on pull request #2727: URL: https://github.com/apache/hadoop/pull/2727#issuecomment-788518502 > > According to the comment in that test case, it "should not time out because effective rpc-timeout is multiple of ping interval: 1600 (= 800 * (1000 / 800 + 1))",

[jira] [Work logged] (HADOOP-17552) The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistaken usage of the rpcTimeout configuratio

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17552?focusedWorklogId=559701=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559701 ] ASF GitHub Bot logged work on HADOOP-17552: --- Author: ASF GitHub Bot

[GitHub] [hadoop] functioner edited a comment on pull request #2727: HADOOP-17552. The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to th

2021-03-01 Thread GitBox
functioner edited a comment on pull request #2727: URL: https://github.com/apache/hadoop/pull/2727#issuecomment-788518502 > > According to the comment in that test case, it "should not time out because effective rpc-timeout is multiple of ping interval: 1600 (= 800 * (1000 / 800 + 1))",

[jira] [Work logged] (HADOOP-17552) The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistaken usage of the rpcTimeout configuratio

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17552?focusedWorklogId=559700=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559700 ] ASF GitHub Bot logged work on HADOOP-17552: --- Author: ASF GitHub Bot

[jira] [Work logged] (HADOOP-17552) The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistaken usage of the rpcTimeout configuratio

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17552?focusedWorklogId=559699=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559699 ] ASF GitHub Bot logged work on HADOOP-17552: --- Author: ASF GitHub Bot

[GitHub] [hadoop] functioner edited a comment on pull request #2727: HADOOP-17552. The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to th

2021-03-01 Thread GitBox
functioner edited a comment on pull request #2727: URL: https://github.com/apache/hadoop/pull/2727#issuecomment-788518502 > > According to the comment in that test case, it "should not time out because effective rpc-timeout is multiple of ping interval: 1600 (= 800 * (1000 / 800 + 1))",

[GitHub] [hadoop] functioner commented on pull request #2727: HADOOP-17552. The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mista

2021-03-01 Thread GitBox
functioner commented on pull request #2727: URL: https://github.com/apache/hadoop/pull/2727#issuecomment-788518502 > > According to the comment in that test case, it "should not time out because effective rpc-timeout is multiple of ping interval: 1600 (= 800 * (1000 / 800 + 1))", and it

[jira] [Work logged] (HADOOP-17552) The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistaken usage of the rpcTimeout configuratio

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17552?focusedWorklogId=559695=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559695 ] ASF GitHub Bot logged work on HADOOP-17552: --- Author: ASF GitHub Bot

[GitHub] [hadoop] functioner edited a comment on pull request #2727: HADOOP-17552. The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to th

2021-03-01 Thread GitBox
functioner edited a comment on pull request #2727: URL: https://github.com/apache/hadoop/pull/2727#issuecomment-788507261 > > According to the comment in that test case, it "should not time out because effective rpc-timeout is multiple of ping interval: 1600 (= 800 * (1000 / 800 + 1))",

[jira] [Work logged] (HADOOP-17552) The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistaken usage of the rpcTimeout configuratio

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17552?focusedWorklogId=559693=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559693 ] ASF GitHub Bot logged work on HADOOP-17552: --- Author: ASF GitHub Bot

[GitHub] [hadoop] functioner commented on pull request #2727: HADOOP-17552. The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mista

2021-03-01 Thread GitBox
functioner commented on pull request #2727: URL: https://github.com/apache/hadoop/pull/2727#issuecomment-788507261 > > According to the comment in that test case, it "should not time out because effective rpc-timeout is multiple of ping interval: 1600 (= 800 * (1000 / 800 + 1))", and it

[jira] [Work logged] (HADOOP-17552) The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistaken usage of the rpcTimeout configuratio

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17552?focusedWorklogId=559659=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559659 ] ASF GitHub Bot logged work on HADOOP-17552: --- Author: ASF GitHub Bot

[GitHub] [hadoop] functioner commented on pull request #2727: HADOOP-17552. The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mista

2021-03-01 Thread GitBox
functioner commented on pull request #2727: URL: https://github.com/apache/hadoop/pull/2727#issuecomment-788419641 > > According to the comment in that test case, it "should not time out because effective rpc-timeout is multiple of ping interval: 1600 (= 800 * (1000 / 800 + 1))", and it

[jira] [Work logged] (HADOOP-17552) The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistaken usage of the rpcTimeout configuratio

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17552?focusedWorklogId=559649=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559649 ] ASF GitHub Bot logged work on HADOOP-17552: --- Author: ASF GitHub Bot

[GitHub] [hadoop] iwasakims commented on pull request #2727: HADOOP-17552. The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistak

2021-03-01 Thread GitBox
iwasakims commented on pull request #2727: URL: https://github.com/apache/hadoop/pull/2727#issuecomment-788413458 > According to the comment in that test case, it "should not time out because effective rpc-timeout is multiple of ping interval: 1600 (= 800 * (1000 / 800 + 1))", and it

[jira] [Created] (HADOOP-17557) skip-dirs option is not processed by Yetus

2021-03-01 Thread Ahmed Hussein (Jira)
Ahmed Hussein created HADOOP-17557: -- Summary: skip-dirs option is not processed by Yetus Key: HADOOP-17557 URL: https://issues.apache.org/jira/browse/HADOOP-17557 Project: Hadoop Common

[jira] [Work logged] (HADOOP-17552) The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistaken usage of the rpcTimeout configuratio

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17552?focusedWorklogId=559447=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559447 ] ASF GitHub Bot logged work on HADOOP-17552: --- Author: ASF GitHub Bot

[GitHub] [hadoop] functioner commented on pull request #2727: HADOOP-17552. The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mista

2021-03-01 Thread GitBox
functioner commented on pull request #2727: URL: https://github.com/apache/hadoop/pull/2727#issuecomment-788083854 > > @functioner see TestRPC#testClientRpcTimeout > > Maybe It is expected that Send ping all the time when _ipc.client.rpc-timeout.ms_ less than 0. > > @iwasakims Do you

[jira] [Work logged] (HADOOP-17552) The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistaken usage of the rpcTimeout configuratio

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17552?focusedWorklogId=559439=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559439 ] ASF GitHub Bot logged work on HADOOP-17552: --- Author: ASF GitHub Bot

[GitHub] [hadoop] functioner commented on pull request #2727: HADOOP-17552. The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mista

2021-03-01 Thread GitBox
functioner commented on pull request #2727: URL: https://github.com/apache/hadoop/pull/2727#issuecomment-788070811 > From the description of [HADOOP-17552](https://issues.apache.org/jira/browse/HADOOP-17552): > > > We are doing some systematic fault injection testing in Hadoop-3.2.2

[GitHub] [hadoop] tomscut commented on pull request #2668: HDFS-15808. Add metrics for FSNamesystem read/write lock hold long time

2021-03-01 Thread GitBox
tomscut commented on pull request #2668: URL: https://github.com/apache/hadoop/pull/2668#issuecomment-788061149 Hi @arp7 , could you take a quick look at this? Let's see if there are any other questions. Thank you. This is

[GitHub] [hadoop] tomscut commented on pull request #2718: HDFS-15854. Make some parameters configurable for SlowDiskTracker and…

2021-03-01 Thread GitBox
tomscut commented on pull request #2718: URL: https://github.com/apache/hadoop/pull/2718#issuecomment-788058754 > Okay I see, the only failure is the two unrelated unit tests. I will merge this shortly. Thank you very much.

[GitHub] [hadoop] arp7 merged pull request #2718: HDFS-15854. Make some parameters configurable for SlowDiskTracker and…

2021-03-01 Thread GitBox
arp7 merged pull request #2718: URL: https://github.com/apache/hadoop/pull/2718 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [hadoop] arp7 commented on pull request #2718: HDFS-15854. Make some parameters configurable for SlowDiskTracker and…

2021-03-01 Thread GitBox
arp7 commented on pull request #2718: URL: https://github.com/apache/hadoop/pull/2718#issuecomment-788056433 Okay I see, the only failure is the two unrelated unit tests. I will merge this shortly. This is an automated

[GitHub] [hadoop] tomscut edited a comment on pull request #2718: HDFS-15854. Make some parameters configurable for SlowDiskTracker and…

2021-03-01 Thread GitBox
tomscut edited a comment on pull request #2718: URL: https://github.com/apache/hadoop/pull/2718#issuecomment-788048147 > Hi @tomscut, I don't see where the Jenkins checks have run successfully. Can you point me to the report? Hi @arp7 , thanks for your review. I saw

[GitHub] [hadoop] tomscut commented on pull request #2718: HDFS-15854. Make some parameters configurable for SlowDiskTracker and…

2021-03-01 Thread GitBox
tomscut commented on pull request #2718: URL: https://github.com/apache/hadoop/pull/2718#issuecomment-788048147 > Hi @tomscut, I don't see where the Jenkins checks have run successfully. Can you point me to the report? Hi @arp7 , thanks for your review. I saw pre-check

[GitHub] [hadoop] arp7 commented on pull request #2718: HDFS-15854. Make some parameters configurable for SlowDiskTracker and…

2021-03-01 Thread GitBox
arp7 commented on pull request #2718: URL: https://github.com/apache/hadoop/pull/2718#issuecomment-788042878 Hi @tomscut, I don't see where the Jenkins checks have run successfully. Can you point me to the report? This is

[GitHub] [hadoop] qizhu-lucas commented on pull request #2721: HDFS-15856: Make recover the pipeline in same packet exceed times for…

2021-03-01 Thread GitBox
qizhu-lucas commented on pull request #2721: URL: https://github.com/apache/hadoop/pull/2721#issuecomment-788040129 I think the test failed is not related to this. This is an automated message from the Apache Git

[GitHub] [hadoop] mukul1987 commented on pull request #2728: HDFS-15865 Interrupt DataStreamer thread if no ack

2021-03-01 Thread GitBox
mukul1987 commented on pull request #2728: URL: https://github.com/apache/hadoop/pull/2728#issuecomment-788011190 @karthikhw , the patch generally looks good to me, However what are the cases when the nodes is null ? This

[GitHub] [hadoop] mukul1987 closed pull request #2728: HDFS-15865 Interrupt DataStreamer thread if no ack

2021-03-01 Thread GitBox
mukul1987 closed pull request #2728: URL: https://github.com/apache/hadoop/pull/2728 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[jira] [Commented] (HADOOP-17532) Yarn Job execution get failed when LZ4 Compression Codec is used

2021-03-01 Thread Bhavik Patel (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17292920#comment-17292920 ] Bhavik Patel commented on HADOOP-17532: --- is the issue here caused by the conflicting version from

[jira] [Work logged] (HADOOP-17471) ABFS to collect IOStatistics

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17471?focusedWorklogId=559400=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559400 ] ASF GitHub Bot logged work on HADOOP-17471: --- Author: ASF GitHub Bot

[GitHub] [hadoop] mehakmeet commented on pull request #2731: HADOOP-17471. ABFS to collect IOStatistics

2021-03-01 Thread GitBox
mehakmeet commented on pull request #2731: URL: https://github.com/apache/hadoop/pull/2731#issuecomment-787996700 Need to remove DurationTrackers from streams, since now I am collecting them from AbfsRestOperation. This is

[jira] [Commented] (HADOOP-17553) FileSystem.close() to optionally log IOStats; save to local dir

2021-03-01 Thread Mehakmeet Singh (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17292910#comment-17292910 ] Mehakmeet Singh commented on HADOOP-17553: -- So, few doubts I had: * Creating JSON is also

[jira] [Updated] (HADOOP-17471) ABFS to collect IOStatistics

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-17471: Labels: pull-request-available (was: ) > ABFS to collect IOStatistics >

[jira] [Work logged] (HADOOP-17471) ABFS to collect IOStatistics

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17471?focusedWorklogId=559393=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559393 ] ASF GitHub Bot logged work on HADOOP-17471: --- Author: ASF GitHub Bot

[GitHub] [hadoop] mehakmeet opened a new pull request #2731: HADOOP-17471. ABFS to collect IOStatistics

2021-03-01 Thread GitBox
mehakmeet opened a new pull request #2731: URL: https://github.com/apache/hadoop/pull/2731 Tested by: mvn -T 16 -Dparallel-tests=abfs clean verify Region: East US ``` [INFO] Results: [INFO] [INFO] Tests run: 95, Failures: 0, Errors: 0, Skipped: 0 ``` ```

[jira] [Created] (HADOOP-17556) Understanding Netty versions and upgrading them (three findings in Hadoop we could upgrade?)

2021-03-01 Thread Adam Roberts (Jira)
Adam Roberts created HADOOP-17556: - Summary: Understanding Netty versions and upgrading them (three findings in Hadoop we could upgrade?) Key: HADOOP-17556 URL: https://issues.apache.org/jira/browse/HADOOP-17556

[jira] [Created] (HADOOP-17555) Image scan shows something in Hadoop using jackson-databind 2.4.0...what?

2021-03-01 Thread Adam Roberts (Jira)
Adam Roberts created HADOOP-17555: - Summary: Image scan shows something in Hadoop using jackson-databind 2.4.0...what? Key: HADOOP-17555 URL: https://issues.apache.org/jira/browse/HADOOP-17555

[GitHub] [hadoop] JJiaguodong opened a new pull request #2730: HDFS-15866. Fix some spelling errors

2021-03-01 Thread GitBox
JJiaguodong opened a new pull request #2730: URL: https://github.com/apache/hadoop/pull/2730 Fix some spelling errors This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [hadoop] Zhangshunyu commented on pull request #2655: HDFS-15714: HDFS Provided Storage Read/Write Mount Support On-the-fly

2021-03-01 Thread GitBox
Zhangshunyu commented on pull request #2655: URL: https://github.com/apache/hadoop/pull/2655#issuecomment-787926710 @PHILO-HE by the way, the read write lock in org/apache/hadoop/hdfs/server/datanode/BlockSender.java:267 `try(AutoCloseableLock lock =

[jira] [Work logged] (HADOOP-17552) The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistaken usage of the rpcTimeout configuratio

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17552?focusedWorklogId=559323=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559323 ] ASF GitHub Bot logged work on HADOOP-17552: --- Author: ASF GitHub Bot

[jira] [Work logged] (HADOOP-17552) The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistaken usage of the rpcTimeout configuratio

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17552?focusedWorklogId=559324=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559324 ] ASF GitHub Bot logged work on HADOOP-17552: --- Author: ASF GitHub Bot

[GitHub] [hadoop] iwasakims edited a comment on pull request #2727: HADOOP-17552. The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the

2021-03-01 Thread GitBox
iwasakims edited a comment on pull request #2727: URL: https://github.com/apache/hadoop/pull/2727#issuecomment-787853012 @functioner @ferhui, I made the timeout configurable by introducing `ipc.client.rpc-timeout.ms` in [HADOOP-11252](https://issues.apache.org/jira/browse/HADOOP-11252).

[GitHub] [hadoop] iwasakims commented on pull request #2727: HADOOP-17552. The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistak

2021-03-01 Thread GitBox
iwasakims commented on pull request #2727: URL: https://github.com/apache/hadoop/pull/2727#issuecomment-787876863 From the description of [HADOOP-17552](https://issues.apache.org/jira/browse/HADOOP-17552): > We are doing some systematic fault injection testing in Hadoop-3.2.2 and

[jira] [Assigned] (HADOOP-17552) The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistaken usage of the rpcTimeout configuration

2021-03-01 Thread Masatake Iwasaki (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki reassigned HADOOP-17552: - Assignee: (was: Masatake Iwasaki) > The read method of

[jira] [Assigned] (HADOOP-17552) The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistaken usage of the rpcTimeout configuration

2021-03-01 Thread Masatake Iwasaki (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki reassigned HADOOP-17552: - Assignee: Masatake Iwasaki > The read method of

[jira] [Updated] (HADOOP-17552) The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistaken usage of the rpcTimeout configuration

2021-03-01 Thread Masatake Iwasaki (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HADOOP-17552: -- Priority: Major (was: Critical) > The read method of

[jira] [Work logged] (HADOOP-17552) The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistaken usage of the rpcTimeout configuratio

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17552?focusedWorklogId=559305=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559305 ] ASF GitHub Bot logged work on HADOOP-17552: --- Author: ASF GitHub Bot

[GitHub] [hadoop] iwasakims commented on pull request #2727: HADOOP-17552. The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistak

2021-03-01 Thread GitBox
iwasakims commented on pull request #2727: URL: https://github.com/apache/hadoop/pull/2727#issuecomment-787854585 I think the long default timeout (kicked by tcp_retries2) surfaced as RM-HA (YARN) issue like [YARN-2578](https://issues.apache.org/jira/browse/YARN-2587) rather than HDFS

[jira] [Work logged] (HADOOP-17552) The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistaken usage of the rpcTimeout configuratio

2021-03-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17552?focusedWorklogId=559303=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559303 ] ASF GitHub Bot logged work on HADOOP-17552: --- Author: ASF GitHub Bot

[GitHub] [hadoop] iwasakims commented on pull request #2727: HADOOP-17552. The read method of hadoop.ipc.Client$Connection$PingInputStream may swallow java.net.SocketTimeoutException due to the mistak

2021-03-01 Thread GitBox
iwasakims commented on pull request #2727: URL: https://github.com/apache/hadoop/pull/2727#issuecomment-787853012 @functioner @ferhui, I made the timeout configurable by introducing `ipc.client.rpc-timeout.ms` in [HADOOP-11252](https://issues.apache.org/jira/browse/HADOOP-11252). Since

[jira] [Updated] (HADOOP-17548) ABFS: Config for Mkdir overwrite

2021-03-01 Thread Sumangala Patki (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumangala Patki updated HADOOP-17548: - Summary: ABFS: Config for Mkdir overwrite (was: ABFS: Change mkdirs default overwrite

[GitHub] [hadoop] sumangala-patki opened a new pull request #2729: Hadoop 17548

2021-03-01 Thread GitBox
sumangala-patki opened a new pull request #2729: URL: https://github.com/apache/hadoop/pull/2729 The call to mkdirs with overwrite set to true results in an overhead set property call (LMT update, etc), which is unnecessary. Moreover, mkdirs on an existing file path returns success. This

[jira] [Updated] (HADOOP-17548) ABFS: Change mkdirs default overwrite to false

2021-03-01 Thread Sumangala Patki (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumangala Patki updated HADOOP-17548: - Description: The call to mkdirs with overwrite set to true results in an overhead set

[GitHub] [hadoop] karthikhw opened a new pull request #2728: HDFS-15865 Interrupt DataStreamer thread if no ack

2021-03-01 Thread GitBox
karthikhw opened a new pull request #2728: URL: https://github.com/apache/hadoop/pull/2728 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above