[GitHub] [hadoop] qizhu-lucas commented on a change in pull request #2721: HDFS-15856: Make recover the pipeline in same packet exceed times for…
qizhu-lucas commented on a change in pull request #2721:
URL: https://github.com/apache/hadoop/pull/2721#discussion_r584034504
##
File path:
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
##
@@ -529,6 +529,7 @@ boolean doWaitForRestart() {
private static final int CONGESTION_BACK_OFF_MAX_TIME_IN_MS =
CONGESTION_BACKOFF_MEAN_TIME_IN_MS * 10;
private int lastCongestionBackoffTime;
+ private int samePacketMaxRecoveryTimes;
Review comment:
Thanks @Hexiaoqiao for patient review, i have fixed it in latest pull
request.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka merged pull request #2725: YARN-10656. Parsing error in CapacityScheduler.md
aajisaka merged pull request #2725: URL: https://github.com/apache/hadoop/pull/2725 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on pull request #2725: YARN-10656. Parsing error in CapacityScheduler.md
aajisaka commented on pull request #2725: URL: https://github.com/apache/hadoop/pull/2725#issuecomment-786997691 The mvnsite failure is in trunk compile tests (i.e. before applying the patch). Merging this. ``` 04:11:53 +--- 04:11:53 | | || trunk Compile Tests 04:11:53 +--- 04:11:53 | +1 | mvninstall | 32m 32s | trunk passed 04:11:53 | -1 | mvnsite | 0m 25s | hadoop-yarn-site in trunk failed. 04:11:53 | +1 |shadedclient | 46m 13s | branch has no errors when building and 04:11:53 | | || testing our client artifacts. 04:11:53 +--- 04:11:53 | | || Patch Compile Tests 04:11:53 +--- 04:11:53 | +1 | mvninstall | 0m 14s | the patch passed 04:11:53 | +1 | blanks | 0m 0s | The patch has no blanks issues. 04:11:53 | +1 | mvnsite | 0m 18s | the patch passed 04:11:53 | +1 |shadedclient | 13m 15s | patch has no errors when building and 04:11:53 | | || testing our client artifacts. ``` Thank you @pbacsko and @qizhu-lucas for your reviews. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] qizhu-lucas commented on pull request #2725: YARN-10656. Parsing error in CapacityScheduler.md
qizhu-lucas commented on pull request #2725: URL: https://github.com/apache/hadoop/pull/2725#issuecomment-786993759 +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Zhangshunyu commented on pull request #2655: HDFS-15714: HDFS Provided Storage Read/Write Mount Support On-the-fly
Zhangshunyu commented on pull request #2655: URL: https://github.com/apache/hadoop/pull/2655#issuecomment-786986457 Some bugs exist on cluster that while reading data, datanode will throw block not found exception but after some retries the block can be read. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jbrennan333 merged pull request #2722: MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase
jbrennan333 merged pull request #2722: URL: https://github.com/apache/hadoop/pull/2722 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17532) Yarn Job execution get failed when LZ4 Compression Codec is used
[
https://issues.apache.org/jira/browse/HADOOP-17532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291888#comment-17291888
]
Chao Sun commented on HADOOP-17532:
---
Sorry for chiming in late here. Yes I agree we should keep the dependency
either provided or optional in case Hadoop users do not need it. Also, is the
issue here caused by the conflicting version from Kafka? and the change on
provided scope is not that related?
> Yarn Job execution get failed when LZ4 Compression Codec is used
>
>
> Key: HADOOP-17532
> URL: https://issues.apache.org/jira/browse/HADOOP-17532
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Bhavik Patel
>Priority: Major
> Attachments: HADOOP-17532.001.patch, LZ4.png, lz4-test.jpg
>
>
> When we try to compress a file using the LZ4 codec compression type then the
> yarn job gets failed with the error message :
> {code:java}
> net.jpountz.lz4.LZ4Compressorcompres(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)V
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] pbacsko commented on pull request #2725: YARN-10656. Parsing error in CapacityScheduler.md
pbacsko commented on pull request #2725: URL: https://github.com/apache/hadoop/pull/2725#issuecomment-786844266 +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17551) Upgrade maven-site-plugin to 3.9.1
[ https://issues.apache.org/jira/browse/HADOOP-17551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-17551: --- Issue Type: Improvement (was: Bug) > Upgrade maven-site-plugin to 3.9.1 > -- > > Key: HADOOP-17551 > URL: https://issues.apache.org/jira/browse/HADOOP-17551 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Akira Ajisaka >Priority: Major > Labels: newbie > > After upgrading maven-site-plugin, error messages will be more detailed and > it will help debugging. > > Ref: > https://issues.apache.org/jira/browse/YARN-10656?focusedCommentId=17291846&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17291846 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17551) Upgrade maven-site-plugin to 3.9.1
Akira Ajisaka created HADOOP-17551: -- Summary: Upgrade maven-site-plugin to 3.9.1 Key: HADOOP-17551 URL: https://issues.apache.org/jira/browse/HADOOP-17551 Project: Hadoop Common Issue Type: Bug Reporter: Akira Ajisaka After upgrading maven-site-plugin, error messages will be more detailed and it will help debugging. Ref: https://issues.apache.org/jira/browse/YARN-10656?focusedCommentId=17291846&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17291846 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on pull request #2725: YARN-10656. Parsing error in CapacityScheduler.md
aajisaka commented on pull request #2725: URL: https://github.com/apache/hadoop/pull/2725#issuecomment-786810071 Built locally after applying the patch. `<` and `>` are escaped as expected.  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka opened a new pull request #2725: YARN-10656. Parsing error in CapacityScheduler.md
aajisaka opened a new pull request #2725: URL: https://github.com/apache/hadoop/pull/2725 JIRA: https://issues.apache.org/jira/browse/YARN-10656 Manually tested. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17545) Provide snapshot builds on Maven central
[ https://issues.apache.org/jira/browse/HADOOP-17545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka resolved HADOOP-17545. Resolution: Done Close this. Thank you [~aroberts] for your report. > Provide snapshot builds on Maven central > > > Key: HADOOP-17545 > URL: https://issues.apache.org/jira/browse/HADOOP-17545 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Adam Roberts >Priority: Minor > > Hey everyone, I'm looking to build the shaded Hadoop/Flink jar using the very > latest Hadoop code that isn't yet in a Maven repository AFAIK (I'm looking at > [https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/).] Entirely > possible that's not the right place... > > Are there plans or is anyone working on, or have I just missed, binaries > being available (say, Hadoop 3.3-snapshot)? > > I remember working on Spark and that was a thing, IIRC you could set a flag > to true in any of the pom.xmls to accept snapshots and not published things. > > It's entirely possible I've just forgot how to do it and can't see it well > documented anywhere, and I don't believe I have to go through the steps of > setting up a Maven repository somewhere (I want to do the build in Docker, > and in the pom.xml I would love to just say: use Hadoop version 3.3-snapshot). > > To give some context, I would like to build the Hadoop/Flink shaded jar using > the yet to be released Hadoop 3.3 branch's code, so I can then go ahead and > security scan that and test it out. > > Thanks in advance! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17532) Yarn Job execution get failed when LZ4 Compression Codec is used
[
https://issues.apache.org/jira/browse/HADOOP-17532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291817#comment-17291817
]
L. C. Hsieh commented on HADOOP-17532:
--
As I said in above comment, it is intentional to have it as provided scope to
avoid pull the dependency for the users not use the compression codec.
> Yarn Job execution get failed when LZ4 Compression Codec is used
>
>
> Key: HADOOP-17532
> URL: https://issues.apache.org/jira/browse/HADOOP-17532
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Bhavik Patel
>Priority: Major
> Attachments: HADOOP-17532.001.patch, LZ4.png, lz4-test.jpg
>
>
> When we try to compress a file using the LZ4 codec compression type then the
> yarn job gets failed with the error message :
> {code:java}
> net.jpountz.lz4.LZ4Compressorcompres(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)V
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] amahussein commented on pull request #2722: MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase
amahussein commented on pull request #2722: URL: https://github.com/apache/hadoop/pull/2722#issuecomment-786748318 I see a previous Jira on Yetus https://issues.apache.org/jira/browse/YETUS-485 @aajisaka Do you have any thoughts about yetus seeming to fail to pick up all the changes after a rebase? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] amahussein commented on pull request #2722: MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase
amahussein commented on pull request #2722: URL: https://github.com/apache/hadoop/pull/2722#issuecomment-786728711 The build error is misleading once more. It complains for missing symbol but the code is already there. The patch compiles successfuly. It seems this happens when I force push several commits after rebasing. We may need to file a separate jira about that. Thanks @jbrennan333 for the review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17532) Yarn Job execution get failed when LZ4 Compression Codec is used
[
https://issues.apache.org/jira/browse/HADOOP-17532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291641#comment-17291641
]
Bhavik Patel commented on HADOOP-17532:
---
Does the another version come from it? ==> Yes, but anyways we need that
dependency so is it better to add it as a part of a deployment.
Any specific reason for keeping provided scope?
> Yarn Job execution get failed when LZ4 Compression Codec is used
>
>
> Key: HADOOP-17532
> URL: https://issues.apache.org/jira/browse/HADOOP-17532
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Bhavik Patel
>Priority: Major
> Attachments: HADOOP-17532.001.patch, LZ4.png, lz4-test.jpg
>
>
> When we try to compress a file using the LZ4 codec compression type then the
> yarn job gets failed with the error message :
> {code:java}
> net.jpountz.lz4.LZ4Compressorcompres(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)V
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17550) property 'ssl.server.keystore.location' has not been set in the ssl configuration file
hamado dene created HADOOP-17550: Summary: property 'ssl.server.keystore.location' has not been set in the ssl configuration file Key: HADOOP-17550 URL: https://issues.apache.org/jira/browse/HADOOP-17550 Project: Hadoop Common Issue Type: Bug Components: conf Affects Versions: 2.8.5 Reporter: hamado dene I trying to install hadoop cluster HA , but datanode does not start properly; I get this errror: 2021-02-23 17:13:26,934 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain java.io.IOException: java.security.GeneralSecurityException: The property 'ssl.server.keystore.location' has not been set in the ssl configuration file. at org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer.(DatanodeHttpServer.java:199) at org.apache.hadoop.hdfs.server.datanode.DataNode.startInfoServer(DataNode.java:905) at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1303) at org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:481) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2609) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2497) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2544) at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2729) at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2753) Caused by: java.security.GeneralSecurityException: The property 'ssl.server.keystore.location' has not been set in the ssl configuration file. at org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory.init(FileBasedKeyStoresFactory.java:152) at org.apache.hadoop.security.ssl.SSLFactory.init(SSLFactory.java:148) at org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer.(DatanodeHttpServer.java:197) ... 8 more But in my ssl-server.xml i correctly set this property: ssl.server.keystore.location /data/hadoop/server.jks Keystore to be used by clients like distcp. Must be specified. ssl.server.keystore.password Optional. Default value is "". ssl.server.keystore.keypassword x Optional. Default value is "". ssl.server.keystore.type jks Optional. The keystore file format, default value is "jks". Do you have any suggestion to solve this problem? my hadoop version is: 2.8.5 java version: 8 SO: centos 7 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on a change in pull request #2721: HDFS-15856: Make recover the pipeline in same packet exceed times for…
Hexiaoqiao commented on a change in pull request #2721:
URL: https://github.com/apache/hadoop/pull/2721#discussion_r583497023
##
File path:
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
##
@@ -529,6 +529,7 @@ boolean doWaitForRestart() {
private static final int CONGESTION_BACK_OFF_MAX_TIME_IN_MS =
CONGESTION_BACKOFF_MEAN_TIME_IN_MS * 10;
private int lastCongestionBackoffTime;
+ private int samePacketMaxRecoveryTimes;
Review comment:
It is better to update this variable and related method name to
`maxPipelineRecoveryRetries`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
