[GitHub] [hadoop] liuml07 commented on a change in pull request #1942: MAPREDUCE-7270. TestHistoryViewerPrinter could be failed when the locale isn't English.
liuml07 commented on a change in pull request #1942: URL: https://github.com/apache/hadoop/pull/1942#discussion_r590030931 ## File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestHistoryViewerPrinter.java ## @@ -43,6 +46,19 @@ private final String LINE_SEPARATOR = System.lineSeparator(); + private static Locale DEFAULT_LOCALE; Review comment: This is only initialized during the static `@BeforeClass setUp()` method, and it will not change during every test cases in this class. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ferhui closed pull request #2746: HDFS-15875. Check whether file is being truncated before truncate
ferhui closed pull request #2746: URL: https://github.com/apache/hadoop/pull/2746 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17570) Apply YETUS-1102 to re-enable GitHub comments
[ https://issues.apache.org/jira/browse/HADOOP-17570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-17570: Labels: pull-request-available (was: ) > Apply YETUS-1102 to re-enable GitHub comments > - > > Key: HADOOP-17570 > URL: https://issues.apache.org/jira/browse/HADOOP-17570 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Yetus 0.13.0 enabled updating GitHub status instead of commenting the report, > however, the report comments are still useful for some cases. Let's apply > YETUS-1102 to re-enable the comments. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17570) Apply YETUS-1102 to re-enable GitHub comments
[ https://issues.apache.org/jira/browse/HADOOP-17570?focusedWorklogId=562860=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-562860 ] ASF GitHub Bot logged work on HADOOP-17570: --- Author: ASF GitHub Bot Created on: 09/Mar/21 07:18 Start Date: 09/Mar/21 07:18 Worklog Time Spent: 10m Work Description: asf-cloudbees-jenkins-ci-hadoop[bot] commented on pull request #2745: URL: https://github.com/apache/hadoop/pull/2745#issuecomment-793476647 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 34s | | Docker mode activated. | | -1 :x: | yetus | 0m 5s | | Unprocessed flag(s): --findbugs-strict-precheck | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2745/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2745 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2745/4/console | | versions | git=2.25.1 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 562860) Remaining Estimate: 0h Time Spent: 10m > Apply YETUS-1102 to re-enable GitHub comments > - > > Key: HADOOP-17570 > URL: https://issues.apache.org/jira/browse/HADOOP-17570 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Yetus 0.13.0 enabled updating GitHub status instead of commenting the report, > however, the report comments are still useful for some cases. Let's apply > YETUS-1102 to re-enable the comments. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] asf-cloudbees-jenkins-ci-hadoop[bot] commented on pull request #2745: HADOOP-17570. Apply YETUS-1102 to re-enable GitHub comments
asf-cloudbees-jenkins-ci-hadoop[bot] commented on pull request #2745: URL: https://github.com/apache/hadoop/pull/2745#issuecomment-793476647 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 34s | | Docker mode activated. | | -1 :x: | yetus | 0m 5s | | Unprocessed flag(s): --findbugs-strict-precheck | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2745/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2745 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2745/4/console | | versions | git=2.25.1 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17570) Apply YETUS-1102 to re-enable GitHub comments
[ https://issues.apache.org/jira/browse/HADOOP-17570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17297884#comment-17297884 ] Akira Ajisaka commented on HADOOP-17570: PR: https://github.com/apache/hadoop/pull/2745 > Apply YETUS-1102 to re-enable GitHub comments > - > > Key: HADOOP-17570 > URL: https://issues.apache.org/jira/browse/HADOOP-17570 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > > Yetus 0.13.0 enabled updating GitHub status instead of commenting the report, > however, the report comments are still useful for some cases. Let's apply > YETUS-1102 to re-enable the comments. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17570) Apply YETUS-1102 to re-enable GitHub comments
[ https://issues.apache.org/jira/browse/HADOOP-17570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-17570: --- Issue Type: Improvement (was: Bug) > Apply YETUS-1102 to re-enable GitHub comments > - > > Key: HADOOP-17570 > URL: https://issues.apache.org/jira/browse/HADOOP-17570 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > > Yetus 0.13.0 enabled updating GitHub status instead of commenting the report, > however, the report comments are still useful for some cases. Let's apply > YETUS-1102 to re-enable the comments. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on pull request #2745: Test YETUS-1102 (Add an option to comment to GitHub PR)
aajisaka commented on pull request #2745: URL: https://github.com/apache/hadoop/pull/2745#issuecomment-793472365 Filed https://issues.apache.org/jira/browse/HADOOP-17570. We need to merge https://github.com/apache/hadoop/pull/2753 before this PR because Yetus dropped findbugs support before YETUS-1102. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17570) Apply YETUS-1102 to re-enable GitHub comments
Akira Ajisaka created HADOOP-17570: -- Summary: Apply YETUS-1102 to re-enable GitHub comments Key: HADOOP-17570 URL: https://issues.apache.org/jira/browse/HADOOP-17570 Project: Hadoop Common Issue Type: Bug Components: build Reporter: Akira Ajisaka Assignee: Akira Ajisaka Yetus 0.13.0 enabled updating GitHub status instead of commenting the report, however, the report comments are still useful for some cases. Let's apply YETUS-1102 to re-enable the comments. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16870) Use spotbugs-maven-plugin instead of findbugs-maven-plugin
[ https://issues.apache.org/jira/browse/HADOOP-16870?focusedWorklogId=562851=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-562851 ] ASF GitHub Bot logged work on HADOOP-16870: --- Author: ASF GitHub Bot Created on: 09/Mar/21 07:00 Start Date: 09/Mar/21 07:00 Worklog Time Spent: 10m Work Description: asf-cloudbees-jenkins-ci-hadoop[bot] commented on pull request #2753: URL: https://github.com/apache/hadoop/pull/2753#issuecomment-793467556 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2753/1/console in case of problems. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 562851) Time Spent: 5h 20m (was: 5h 10m) > Use spotbugs-maven-plugin instead of findbugs-maven-plugin > -- > > Key: HADOOP-16870 > URL: https://issues.apache.org/jira/browse/HADOOP-16870 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 5h 20m > Remaining Estimate: 0h > > findbugs-maven-plugin is no longer maintained. Use spotbugs-maven-plugin > instead. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] asf-cloudbees-jenkins-ci-hadoop[bot] commented on pull request #2753: HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin
asf-cloudbees-jenkins-ci-hadoop[bot] commented on pull request #2753: URL: https://github.com/apache/hadoop/pull/2753#issuecomment-793467556 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2753/1/console in case of problems. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16870) Use spotbugs-maven-plugin instead of findbugs-maven-plugin
[ https://issues.apache.org/jira/browse/HADOOP-16870?focusedWorklogId=562847=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-562847 ] ASF GitHub Bot logged work on HADOOP-16870: --- Author: ASF GitHub Bot Created on: 09/Mar/21 06:58 Start Date: 09/Mar/21 06:58 Worklog Time Spent: 10m Work Description: aajisaka opened a new pull request #2753: URL: https://github.com/apache/hadoop/pull/2753 JIRA: https://issues.apache.org/jira/browse/HADOOP-16870 In addition to #2454, upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0 to fix https://github.com/spotbugs/spotbugs/issues/1161 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 562847) Time Spent: 5h 10m (was: 5h) > Use spotbugs-maven-plugin instead of findbugs-maven-plugin > -- > > Key: HADOOP-16870 > URL: https://issues.apache.org/jira/browse/HADOOP-16870 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 5h 10m > Remaining Estimate: 0h > > findbugs-maven-plugin is no longer maintained. Use spotbugs-maven-plugin > instead. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16870) Use spotbugs-maven-plugin instead of findbugs-maven-plugin
[ https://issues.apache.org/jira/browse/HADOOP-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-16870: --- Status: Patch Available (was: Reopened) > Use spotbugs-maven-plugin instead of findbugs-maven-plugin > -- > > Key: HADOOP-16870 > URL: https://issues.apache.org/jira/browse/HADOOP-16870 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 5h 10m > Remaining Estimate: 0h > > findbugs-maven-plugin is no longer maintained. Use spotbugs-maven-plugin > instead. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka opened a new pull request #2753: HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin
aajisaka opened a new pull request #2753: URL: https://github.com/apache/hadoop/pull/2753 JIRA: https://issues.apache.org/jira/browse/HADOOP-16870 In addition to #2454, upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0 to fix https://github.com/spotbugs/spotbugs/issues/1161 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17563) Update Bouncy Castle to 1.68
[ https://issues.apache.org/jira/browse/HADOOP-17563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17297856#comment-17297856 ] Takanobu Asanuma commented on HADOOP-17563: --- Sorry, Bouncy Castle 1.68 does not fix the vulnerability. There is not any fixed version for now. The vulnerability may affect you only when you use BKS-V1. Therefore, most Hadoop user won't be affected. > Update Bouncy Castle to 1.68 > > > Key: HADOOP-17563 > URL: https://issues.apache.org/jira/browse/HADOOP-17563 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0, 3.2.3 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Bouncy Castle 1.60 has Hash Collision Vulnerability. Let's update to 1.68. > https://www.sourceclear.com/vulnerability-database/security/hash-collision/java/sid-6009 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16721) Improve S3A rename resilience
[
https://issues.apache.org/jira/browse/HADOOP-16721?focusedWorklogId=562838=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-562838
]
ASF GitHub Bot logged work on HADOOP-16721:
---
Author: ASF GitHub Bot
Created on: 09/Mar/21 05:38
Start Date: 09/Mar/21 05:38
Worklog Time Spent: 10m
Work Description: iwasakims commented on a change in pull request #2742:
URL: https://github.com/apache/hadoop/pull/2742#discussion_r589957067
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestlRenameDeleteRace.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import java.io.IOException;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+
+import com.amazonaws.AmazonClientException;
+import
org.apache.hadoop.thirdparty.com.google.common.util.concurrent.ListeningExecutorService;
+import org.assertj.core.api.Assertions;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.contract.ContractTestUtils;
+import org.apache.hadoop.fs.s3a.AbstractS3ATestBase;
+import org.apache.hadoop.fs.s3a.S3AFileSystem;
+import org.apache.hadoop.util.BlockingThreadPoolExecutorService;
+
+import static org.apache.hadoop.fs.s3a.Constants.DIRECTORY_MARKER_POLICY;
+import static
org.apache.hadoop.fs.s3a.Constants.DIRECTORY_MARKER_POLICY_DELETE;
+import static org.apache.hadoop.fs.s3a.Constants.S3GUARD_METASTORE_NULL;
+import static org.apache.hadoop.fs.s3a.Constants.S3_METADATA_STORE_IMPL;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.getTestBucketName;
+import static
org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides;
+import static org.apache.hadoop.fs.s3a.impl.CallableSupplier.submit;
+import static org.apache.hadoop.fs.s3a.impl.CallableSupplier.waitForCompletion;
+import static org.apache.hadoop.io.IOUtils.cleanupWithLogger;
+
+/**
+ * HADOOP-16721: race condition with delete and rename underneath the same
destination
+ * directory.
+ * This test suite recreates the failure using semaphores to guarantee the
failure
+ * condition is encountered -then verifies that the rename operation is
successful.
+ */
+public class ITestlRenameDeleteRace extends AbstractS3ATestBase {
Review comment:
Is the class name intended to be ITestRenameDeleteRace?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 562838)
Time Spent: 50m (was: 40m)
> Improve S3A rename resilience
> -
>
> Key: HADOOP-16721
> URL: https://issues.apache.org/jira/browse/HADOOP-16721
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> h3. race condition in delete/rename overlap
> If you have multiple threads on a system doing rename operations, then one
> thread doing a delete(dest/subdir) may delete the last file under a subdir,
> and, before its listed and recreated any parent dir marker -other threads may
> conclude there's an empty dest dir and fail.
> This is most likely on an overloaded system with many threads executing
> rename operations, as with parallel copying taking place there are many
> threads to schedule and https connections to pool.
> h3. failure reporting
> the classic \{[rename(source, dest)}} operation returns \{{false}} on certain
>
[GitHub] [hadoop] iwasakims commented on a change in pull request #2742: HADOOP-16721. Improve S3A rename resilience
iwasakims commented on a change in pull request #2742:
URL: https://github.com/apache/hadoop/pull/2742#discussion_r589957067
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestlRenameDeleteRace.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import java.io.IOException;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+
+import com.amazonaws.AmazonClientException;
+import
org.apache.hadoop.thirdparty.com.google.common.util.concurrent.ListeningExecutorService;
+import org.assertj.core.api.Assertions;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.contract.ContractTestUtils;
+import org.apache.hadoop.fs.s3a.AbstractS3ATestBase;
+import org.apache.hadoop.fs.s3a.S3AFileSystem;
+import org.apache.hadoop.util.BlockingThreadPoolExecutorService;
+
+import static org.apache.hadoop.fs.s3a.Constants.DIRECTORY_MARKER_POLICY;
+import static
org.apache.hadoop.fs.s3a.Constants.DIRECTORY_MARKER_POLICY_DELETE;
+import static org.apache.hadoop.fs.s3a.Constants.S3GUARD_METASTORE_NULL;
+import static org.apache.hadoop.fs.s3a.Constants.S3_METADATA_STORE_IMPL;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.getTestBucketName;
+import static
org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides;
+import static org.apache.hadoop.fs.s3a.impl.CallableSupplier.submit;
+import static org.apache.hadoop.fs.s3a.impl.CallableSupplier.waitForCompletion;
+import static org.apache.hadoop.io.IOUtils.cleanupWithLogger;
+
+/**
+ * HADOOP-16721: race condition with delete and rename underneath the same
destination
+ * directory.
+ * This test suite recreates the failure using semaphores to guarantee the
failure
+ * condition is encountered -then verifies that the rename operation is
successful.
+ */
+public class ITestlRenameDeleteRace extends AbstractS3ATestBase {
Review comment:
Is the class name intended to be ITestRenameDeleteRace?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16721) Improve S3A rename resilience
[
https://issues.apache.org/jira/browse/HADOOP-16721?focusedWorklogId=562837=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-562837
]
ASF GitHub Bot logged work on HADOOP-16721:
---
Author: ASF GitHub Bot
Created on: 09/Mar/21 05:37
Start Date: 09/Mar/21 05:37
Worklog Time Spent: 10m
Work Description: iwasakims commented on a change in pull request #2742:
URL: https://github.com/apache/hadoop/pull/2742#discussion_r589956618
##
File path:
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md
##
@@ -1126,6 +1126,26 @@ We also recommend using applications/application
options which do not rename files when committing work or when copying data
to S3, but instead write directly to the final destination.
+## Rename not behaving as "expected"
+
+S3 is not a filesystem. The S3A connector mimics rename by
+
+* HEAD then LIST of source path
+* HEAD then LIST of destination path
+* File-by-file copy of source objects to destination.
+ Parallelized, with page listings of directory objects and issuing of DELETE
requests.
+* Post-delete recreation of destination parent directory marker, if needed.
Review comment:
recreation of **source** parent directory marker?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 562837)
Time Spent: 40m (was: 0.5h)
> Improve S3A rename resilience
> -
>
> Key: HADOOP-16721
> URL: https://issues.apache.org/jira/browse/HADOOP-16721
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h3. race condition in delete/rename overlap
> If you have multiple threads on a system doing rename operations, then one
> thread doing a delete(dest/subdir) may delete the last file under a subdir,
> and, before its listed and recreated any parent dir marker -other threads may
> conclude there's an empty dest dir and fail.
> This is most likely on an overloaded system with many threads executing
> rename operations, as with parallel copying taking place there are many
> threads to schedule and https connections to pool.
> h3. failure reporting
> the classic \{[rename(source, dest)}} operation returns \{{false}} on certain
> failures, which, while somewhat consistent with the posix APIs, turns out to
> be useless for identifying the cause of problems. Applications tend to have
> code which goes
> {code}
> if (!fs.rename(src, dest)) throw new IOException("rename failed");
> {code}
> While ultimately the rename/3 call needs to be made public (HADOOP-11452) it
> would then need a adoption across applications. We can do this in the hadoop
> modules, but for Hive, Spark etc it will take along time.
> Proposed: a switch to tell S3A to stop downgrading certain failures (source
> is dir, dest is file, src==dest, etc) into "false". This can be turned on
> when trying to diagnose why things like Hive are failing.
> Production code: trivial
> * change in rename(),
> * new option
> * docs.
> Test code:
> * need to clear this option for rename contract tests
> * need to create a new FS with this set to verify the various failure modes
> trigger it.
>
> If this works we should do the same for ABFS, GCS. Hey, maybe even HDFS
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] iwasakims commented on a change in pull request #2742: HADOOP-16721. Improve S3A rename resilience
iwasakims commented on a change in pull request #2742: URL: https://github.com/apache/hadoop/pull/2742#discussion_r589956618 ## File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md ## @@ -1126,6 +1126,26 @@ We also recommend using applications/application options which do not rename files when committing work or when copying data to S3, but instead write directly to the final destination. +## Rename not behaving as "expected" + +S3 is not a filesystem. The S3A connector mimics rename by + +* HEAD then LIST of source path +* HEAD then LIST of destination path +* File-by-file copy of source objects to destination. + Parallelized, with page listings of directory objects and issuing of DELETE requests. +* Post-delete recreation of destination parent directory marker, if needed. Review comment: recreation of **source** parent directory marker? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17531) DistCp: Reduce memory usage on copying huge directories
[ https://issues.apache.org/jira/browse/HADOOP-17531?focusedWorklogId=562808=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-562808 ] ASF GitHub Bot logged work on HADOOP-17531: --- Author: ASF GitHub Bot Created on: 09/Mar/21 04:13 Start Date: 09/Mar/21 04:13 Worklog Time Spent: 10m Work Description: ayushtkn commented on pull request #2732: URL: https://github.com/apache/hadoop/pull/2732#issuecomment-793357471 Thanx @jojochuang for the review, I have addressed the review comments, Please have a look. :-) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 562808) Time Spent: 1h 20m (was: 1h 10m) > DistCp: Reduce memory usage on copying huge directories > --- > > Key: HADOOP-17531 > URL: https://issues.apache.org/jira/browse/HADOOP-17531 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Critical > Labels: pull-request-available > Attachments: MoveToStackIterator.patch, gc-NewD-512M-3.8ML.log > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Presently distCp, uses the producer-consumer kind of setup while building the > listing, the input queue and output queue are both unbounded, thus the > listStatus grows quite huge. > Rel Code Part : > https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java#L635 > This goes on bredth-first traversal kind of stuff(uses queue instead of > earlier stack), so if you have files at lower depth, it will like open up the > entire tree and the start processing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on pull request #2732: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories.
ayushtkn commented on pull request #2732: URL: https://github.com/apache/hadoop/pull/2732#issuecomment-793357471 Thanx @jojochuang for the review, I have addressed the review comments, Please have a look. :-) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17531) DistCp: Reduce memory usage on copying huge directories
[
https://issues.apache.org/jira/browse/HADOOP-17531?focusedWorklogId=562807=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-562807
]
ASF GitHub Bot logged work on HADOOP-17531:
---
Author: ASF GitHub Bot
Created on: 09/Mar/21 04:12
Start Date: 09/Mar/21 04:12
Worklog Time Spent: 10m
Work Description: ayushtkn commented on a change in pull request #2732:
URL: https://github.com/apache/hadoop/pull/2732#discussion_r589927863
##
File path:
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java
##
@@ -688,6 +696,71 @@ private void traverseDirectory(SequenceFile.Writer
fileListWriter,
workers.shutdown();
}
+ private void traverseDirectoryLegacy(SequenceFile.Writer fileListWriter,
+ FileSystem sourceFS, ArrayList sourceDirs,
+ Path sourcePathRoot, DistCpContext context, HashSet excludeList,
+ List fileStatuses) throws IOException {
+final boolean preserveAcls = context.shouldPreserve(FileAttribute.ACL);
+final boolean preserveXAttrs = context.shouldPreserve(FileAttribute.XATTR);
+final boolean preserveRawXattrs = context.shouldPreserveRawXattrs();
+
+Stack pathStack = new Stack();
+for (FileStatus fs : sourceDirs) {
+ if (excludeList == null || !excludeList
+ .contains(fs.getPath().toUri().getPath())) {
+pathStack.add(fs);
+ }
+}
+
+while (!pathStack.isEmpty()) {
+ prepareListing(pathStack.pop().getPath(), fileListWriter, sourceFS,
+ sourcePathRoot, context, excludeList, fileStatuses, preserveAcls,
+ preserveXAttrs, preserveRawXattrs);
+}
+ }
+
+ @SuppressWarnings("checkstyle:parameternumber")
+ private void prepareListing(Path path, SequenceFile.Writer fileListWriter,
Review comment:
Done, Refactored `TraverseDirectory` to a class
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 562807)
Time Spent: 1h 10m (was: 1h)
> DistCp: Reduce memory usage on copying huge directories
> ---
>
> Key: HADOOP-17531
> URL: https://issues.apache.org/jira/browse/HADOOP-17531
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Critical
> Labels: pull-request-available
> Attachments: MoveToStackIterator.patch, gc-NewD-512M-3.8ML.log
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Presently distCp, uses the producer-consumer kind of setup while building the
> listing, the input queue and output queue are both unbounded, thus the
> listStatus grows quite huge.
> Rel Code Part :
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java#L635
> This goes on bredth-first traversal kind of stuff(uses queue instead of
> earlier stack), so if you have files at lower depth, it will like open up the
> entire tree and the start processing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17565) Job will hang when disk for logging is full
[ https://issues.apache.org/jira/browse/HADOOP-17565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Zhong updated HADOOP-17565: -- Summary: Job will hang when disk for logging is full (was: Job will hang forever when disk for logging is full) > Job will hang when disk for logging is full > --- > > Key: HADOOP-17565 > URL: https://issues.apache.org/jira/browse/HADOOP-17565 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Affects Versions: 3.3.0 > Environment: Hadoop-3.3.0 >Reporter: Li Zhong >Priority: Major > > When we put the log directory of Hadoop on a devoted disk space and the disk > is closed to full. The job we run will suspend and never abort. > The job we run is 'share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.0.jar' > > From jps, seems everything is running normally > hadoop@lily-OptiPlex-3070:~/hadoop-3.3.0$ jps > 23409 NameNode > 23889 SecondaryNameNode > 24338 NodeManager > 23622 DataNode > 14408 RunJar > 29098 HRegionServer > 14526 Jps > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17531) DistCp: Reduce memory usage on copying huge directories
[ https://issues.apache.org/jira/browse/HADOOP-17531?focusedWorklogId=562806=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-562806 ] ASF GitHub Bot logged work on HADOOP-17531: --- Author: ASF GitHub Bot Created on: 09/Mar/21 04:12 Start Date: 09/Mar/21 04:12 Worklog Time Spent: 10m Work Description: ayushtkn commented on a change in pull request #2732: URL: https://github.com/apache/hadoop/pull/2732#discussion_r589927746 ## File path: hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm ## @@ -362,6 +362,7 @@ Command Line Options | `-copybuffersize ` | Size of the copy buffer to use. By default, `` is set to 8192B | | | `-xtrack ` | Save information about missing source files to the specified path. | This option is only valid with `-update` option. This is an experimental property and it cannot be used with `-atomic` option. | | `-direct` | Write directly to destination paths | Useful for avoiding potentially very expensive temporary file rename operations when the destination is an object store | +| `-useIterator` | Uses single threaded listStatusIterator to build listing | Useful for saving memory at the client side. | Review comment: I have updated the document, Let me know if something more can be improved. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 562806) Time Spent: 1h (was: 50m) > DistCp: Reduce memory usage on copying huge directories > --- > > Key: HADOOP-17531 > URL: https://issues.apache.org/jira/browse/HADOOP-17531 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Critical > Labels: pull-request-available > Attachments: MoveToStackIterator.patch, gc-NewD-512M-3.8ML.log > > Time Spent: 1h > Remaining Estimate: 0h > > Presently distCp, uses the producer-consumer kind of setup while building the > listing, the input queue and output queue are both unbounded, thus the > listStatus grows quite huge. > Rel Code Part : > https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java#L635 > This goes on bredth-first traversal kind of stuff(uses queue instead of > earlier stack), so if you have files at lower depth, it will like open up the > entire tree and the start processing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #2732: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories.
ayushtkn commented on a change in pull request #2732:
URL: https://github.com/apache/hadoop/pull/2732#discussion_r589927863
##
File path:
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java
##
@@ -688,6 +696,71 @@ private void traverseDirectory(SequenceFile.Writer
fileListWriter,
workers.shutdown();
}
+ private void traverseDirectoryLegacy(SequenceFile.Writer fileListWriter,
+ FileSystem sourceFS, ArrayList sourceDirs,
+ Path sourcePathRoot, DistCpContext context, HashSet excludeList,
+ List fileStatuses) throws IOException {
+final boolean preserveAcls = context.shouldPreserve(FileAttribute.ACL);
+final boolean preserveXAttrs = context.shouldPreserve(FileAttribute.XATTR);
+final boolean preserveRawXattrs = context.shouldPreserveRawXattrs();
+
+Stack pathStack = new Stack();
+for (FileStatus fs : sourceDirs) {
+ if (excludeList == null || !excludeList
+ .contains(fs.getPath().toUri().getPath())) {
+pathStack.add(fs);
+ }
+}
+
+while (!pathStack.isEmpty()) {
+ prepareListing(pathStack.pop().getPath(), fileListWriter, sourceFS,
+ sourcePathRoot, context, excludeList, fileStatuses, preserveAcls,
+ preserveXAttrs, preserveRawXattrs);
+}
+ }
+
+ @SuppressWarnings("checkstyle:parameternumber")
+ private void prepareListing(Path path, SequenceFile.Writer fileListWriter,
Review comment:
Done, Refactored `TraverseDirectory` to a class
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #2732: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories.
ayushtkn commented on a change in pull request #2732: URL: https://github.com/apache/hadoop/pull/2732#discussion_r589927746 ## File path: hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm ## @@ -362,6 +362,7 @@ Command Line Options | `-copybuffersize ` | Size of the copy buffer to use. By default, `` is set to 8192B | | | `-xtrack ` | Save information about missing source files to the specified path. | This option is only valid with `-update` option. This is an experimental property and it cannot be used with `-atomic` option. | | `-direct` | Write directly to destination paths | Useful for avoiding potentially very expensive temporary file rename operations when the destination is an object store | +| `-useIterator` | Uses single threaded listStatusIterator to build listing | Useful for saving memory at the client side. | Review comment: I have updated the document, Let me know if something more can be improved. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17557) skip-dir option is not processed by Yetus
[
https://issues.apache.org/jira/browse/HADOOP-17557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-17557:
---
Fix Version/s: 3.2.3
2.10.2
3.1.5
3.4.0
3.3.1
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to all the active branches. Thank you [~ahussein]!
> skip-dir option is not processed by Yetus
> -
>
> Key: HADOOP-17557
> URL: https://issues.apache.org/jira/browse/HADOOP-17557
> Project: Hadoop Common
> Issue Type: Bug
> Components: build, precommit, yetus
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Fix For: 3.3.1, 3.4.0, 3.1.5, 2.10.2, 3.2.3
>
> Attachments: HADOOP-17557.001.patch
>
>
> Running test patch locally does not work anymore after the Yetus upgrade
> {code:bash}
> dev-support/bin/test-patch --plugins="maven,checkstyle" --test-parallel=true
> patch-file.patch
> {code}
> Error is
> {code:bash}
> Testing patch on trunk.
> ERROR: Unprocessed flag(s): --skip-dir
> environment {
> SOURCEDIR = 'src'
> // will also need to change notification section below
> PATCHDIR = 'out'
> DOCKERFILE = "${SOURCEDIR}/dev-support/docker/Dockerfile"
> YETUS='yetus'
> // Branch or tag name. Yetus release tags are 'rel/X.Y.Z'
> YETUS_VERSION='rel/0.13.0'
> /skip-
> # URL for user-side presentation in reports and such
> to our artifacts
> _ _ __
> | ___|_ _(_) |_ _ _ __ ___| |
> | |_ / _` | | | | | | '__/ _ \ |
> | _| (_| | | | |_| | | | __/_|
> |_| \__,_|_|_|\__,_|_| \___(_)
> | Vote |Subsystem | Runtime | Comment
>
> | -1 | yetus | 0m 05s | Unprocessed flag(s): --skip-dir
> {code}
> It seems that the "{{--skip-dir}}" option supported Yetus release prior to
> 0.11.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17557) skip-dir option is not processed by Yetus
[
https://issues.apache.org/jira/browse/HADOOP-17557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17297780#comment-17297780
]
Akira Ajisaka commented on HADOOP-17557:
+1
> skip-dir option is not processed by Yetus
> -
>
> Key: HADOOP-17557
> URL: https://issues.apache.org/jira/browse/HADOOP-17557
> Project: Hadoop Common
> Issue Type: Bug
> Components: build, precommit, yetus
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: HADOOP-17557.001.patch
>
>
> Running test patch locally does not work anymore after the Yetus upgrade
> {code:bash}
> dev-support/bin/test-patch --plugins="maven,checkstyle" --test-parallel=true
> patch-file.patch
> {code}
> Error is
> {code:bash}
> Testing patch on trunk.
> ERROR: Unprocessed flag(s): --skip-dir
> environment {
> SOURCEDIR = 'src'
> // will also need to change notification section below
> PATCHDIR = 'out'
> DOCKERFILE = "${SOURCEDIR}/dev-support/docker/Dockerfile"
> YETUS='yetus'
> // Branch or tag name. Yetus release tags are 'rel/X.Y.Z'
> YETUS_VERSION='rel/0.13.0'
> /skip-
> # URL for user-side presentation in reports and such
> to our artifacts
> _ _ __
> | ___|_ _(_) |_ _ _ __ ___| |
> | |_ / _` | | | | | | '__/ _ \ |
> | _| (_| | | | |_| | | | __/_|
> |_| \__,_|_|_|\__,_|_| \___(_)
> | Vote |Subsystem | Runtime | Comment
>
> | -1 | yetus | 0m 05s | Unprocessed flag(s): --skip-dir
> {code}
> It seems that the "{{--skip-dir}}" option supported Yetus release prior to
> 0.11.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ferhui commented on a change in pull request #2746: HDFS-15875. Check whether file is being truncated before truncate
ferhui commented on a change in pull request #2746:
URL: https://github.com/apache/hadoop/pull/2746#discussion_r589878331
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
##
@@ -631,10 +698,10 @@ public void testTruncateFailure() throws IOException {
{
try {
fs.truncate(p, 0);
-fail("Truncate must fail since a trancate is already in pregress.");
+fail("Truncate must fail since a truncate is already in progress.");
} catch (IOException expected) {
GenericTestUtils.assertExceptionContains(
-"Failed to TRUNCATE_FILE", expected);
+"/dir/testTruncateFailure is being truncated", expected);
}
Review comment:
@ayushtkn Thanks for review!
This case happens with the same lease holder when enter
recoverLeaseInternal. And our fix throw exception before enter
recoverLeaseInternal. Our UT has 2 clients with different client names.
Will upload fix to throw the same exception.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on pull request #2743: HDFS-15873. Add namenode address in logs for block report
tomscut commented on pull request #2743: URL: https://github.com/apache/hadoop/pull/2743#issuecomment-793228577 Thanks @dineshchitlangia for the commit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16721) Improve S3A rename resilience
[
https://issues.apache.org/jira/browse/HADOOP-16721?focusedWorklogId=562556=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-562556
]
ASF GitHub Bot logged work on HADOOP-16721:
---
Author: ASF GitHub Bot
Created on: 08/Mar/21 19:55
Start Date: 08/Mar/21 19:55
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2742:
URL: https://github.com/apache/hadoop/pull/2742#issuecomment-793030939
Testing: s3 london with markers==keep and delete, s3guard on and off.
* change in error reporting of s3a FS matched with relevant changes in
s3a.xml for contract tests.
* skip tests verifying that you can't rename 2+ levels under a file.
* Failures related to endpoints of common-crawl and
ITestAssumeRole.testAssumeRoleBadInnerAuth: known and fixed in #2675.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 562556)
Time Spent: 0.5h (was: 20m)
> Improve S3A rename resilience
> -
>
> Key: HADOOP-16721
> URL: https://issues.apache.org/jira/browse/HADOOP-16721
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> h3. race condition in delete/rename overlap
> If you have multiple threads on a system doing rename operations, then one
> thread doing a delete(dest/subdir) may delete the last file under a subdir,
> and, before its listed and recreated any parent dir marker -other threads may
> conclude there's an empty dest dir and fail.
> This is most likely on an overloaded system with many threads executing
> rename operations, as with parallel copying taking place there are many
> threads to schedule and https connections to pool.
> h3. failure reporting
> the classic \{[rename(source, dest)}} operation returns \{{false}} on certain
> failures, which, while somewhat consistent with the posix APIs, turns out to
> be useless for identifying the cause of problems. Applications tend to have
> code which goes
> {code}
> if (!fs.rename(src, dest)) throw new IOException("rename failed");
> {code}
> While ultimately the rename/3 call needs to be made public (HADOOP-11452) it
> would then need a adoption across applications. We can do this in the hadoop
> modules, but for Hive, Spark etc it will take along time.
> Proposed: a switch to tell S3A to stop downgrading certain failures (source
> is dir, dest is file, src==dest, etc) into "false". This can be turned on
> when trying to diagnose why things like Hive are failing.
> Production code: trivial
> * change in rename(),
> * new option
> * docs.
> Test code:
> * need to clear this option for rename contract tests
> * need to create a new FS with this set to verify the various failure modes
> trigger it.
>
> If this works we should do the same for ABFS, GCS. Hey, maybe even HDFS
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2742: HADOOP-16721. Improve S3A rename resilience
steveloughran commented on pull request #2742: URL: https://github.com/apache/hadoop/pull/2742#issuecomment-793030939 Testing: s3 london with markers==keep and delete, s3guard on and off. * change in error reporting of s3a FS matched with relevant changes in s3a.xml for contract tests. * skip tests verifying that you can't rename 2+ levels under a file. * Failures related to endpoints of common-crawl and ITestAssumeRole.testAssumeRoleBadInnerAuth: known and fixed in #2675. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17557) skip-dir option is not processed by Yetus
[
https://issues.apache.org/jira/browse/HADOOP-17557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17297586#comment-17297586
]
Hadoop QA commented on HADOOP-17557:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
41s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 53s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} There were no new shellcheck
issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
15s{color} | {color:green}{color} | {color:green} There were no new shelldocs
issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 33s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} || ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green}{color} | {color:green} The patch does not generate
ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 29s{color} |
{color:black}{color} | {color:black}{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/PreCommit-HADOOP-Build/160/artifact/out/Dockerfile
|
| JIRA Issue | HADOOP-17557 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13021882/HADOOP-17557.001.patch
|
| Optional Tests | dupname asflicense shellcheck shelldocs |
| uname | Linux a12792d72a89 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23
18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 6a922f98967 |
| Max. process+thread count | 597 (vs. ulimit of 5500) |
| modules | C: . U: . |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-HADOOP-Build/160/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> skip-dir option is not processed by Yetus
> -
>
> Key: HADOOP-17557
> URL: https://issues.apache.org/jira/browse/HADOOP-17557
> Project: Hadoop Common
> Issue Type: Bug
> Components: build, precommit, yetus
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: HADOOP-17557.001.patch
>
>
> Running test patch locally does not work anymore after the Yetus upgrade
> {code:bash}
> dev-support/bin/test-patch --plugins="maven,checkstyle" --test-parallel=true
> patch-file.patch
> {code}
> Error is
> {code:bash}
> Testing patch on trunk.
> ERROR: Unprocessed flag(s): --skip-dir
> environment {
> SOURCEDIR = 'src'
> // will also need to change notification section below
> PATCHDIR = 'out'
> DOCKERFILE = "${SOURCEDIR}/dev-support/docker/Dockerfile"
> YETUS='yetus'
> // Branch or tag name. Yetus release tags are 'rel/X.Y.Z'
> YETUS_VERSION='rel/0.13.0'
> /skip-
> # URL for user-side presentation in reports and such
> to our artifacts
> _ _ __
> | ___|_ _(_) |_ _ _ __ ___| |
> | |_ / _` | | | | | | '__/ _ \ |
> | _| (_| | | | |_| | | | __/_|
> |_| \__,_|_|_|\__,_|_| \___(_)
> | Vote |Subsystem | Runtime | Comment
>
> | -1 | yetus | 0m 05s | Unprocessed flag(s): --skip-dir
> {code}
> It seems that the "{{--skip-dir}}" option supported Yetus release prior to
> 0.11.
--
This message was sent by
[jira] [Commented] (HADOOP-17557) skip-dir option is not processed by Yetus
[
https://issues.apache.org/jira/browse/HADOOP-17557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17297556#comment-17297556
]
Ahmed Hussein commented on HADOOP-17557:
It was a simple and straightforward fix.
{{dev-support/bin/test-patch.sh}} line 18 had the flag {{--skip-dir}}, instead
of {{--skip-dirs}}.
[~aajisaka], can you please take a look at the patch?
> skip-dir option is not processed by Yetus
> -
>
> Key: HADOOP-17557
> URL: https://issues.apache.org/jira/browse/HADOOP-17557
> Project: Hadoop Common
> Issue Type: Bug
> Components: build, precommit, yetus
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: HADOOP-17557.001.patch
>
>
> Running test patch locally does not work anymore after the Yetus upgrade
> {code:bash}
> dev-support/bin/test-patch --plugins="maven,checkstyle" --test-parallel=true
> patch-file.patch
> {code}
> Error is
> {code:bash}
> Testing patch on trunk.
> ERROR: Unprocessed flag(s): --skip-dir
> environment {
> SOURCEDIR = 'src'
> // will also need to change notification section below
> PATCHDIR = 'out'
> DOCKERFILE = "${SOURCEDIR}/dev-support/docker/Dockerfile"
> YETUS='yetus'
> // Branch or tag name. Yetus release tags are 'rel/X.Y.Z'
> YETUS_VERSION='rel/0.13.0'
> /skip-
> # URL for user-side presentation in reports and such
> to our artifacts
> _ _ __
> | ___|_ _(_) |_ _ _ __ ___| |
> | |_ / _` | | | | | | '__/ _ \ |
> | _| (_| | | | |_| | | | __/_|
> |_| \__,_|_|_|\__,_|_| \___(_)
> | Vote |Subsystem | Runtime | Comment
>
> | -1 | yetus | 0m 05s | Unprocessed flag(s): --skip-dir
> {code}
> It seems that the "{{--skip-dir}}" option supported Yetus release prior to
> 0.11.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17557) skip-dir option is not processed by Yetus
[
https://issues.apache.org/jira/browse/HADOOP-17557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated HADOOP-17557:
---
Attachment: HADOOP-17557.001.patch
Status: Patch Available (was: In Progress)
> skip-dir option is not processed by Yetus
> -
>
> Key: HADOOP-17557
> URL: https://issues.apache.org/jira/browse/HADOOP-17557
> Project: Hadoop Common
> Issue Type: Bug
> Components: build, precommit, yetus
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: HADOOP-17557.001.patch
>
>
> Running test patch locally does not work anymore after the Yetus upgrade
> {code:bash}
> dev-support/bin/test-patch --plugins="maven,checkstyle" --test-parallel=true
> patch-file.patch
> {code}
> Error is
> {code:bash}
> Testing patch on trunk.
> ERROR: Unprocessed flag(s): --skip-dir
> environment {
> SOURCEDIR = 'src'
> // will also need to change notification section below
> PATCHDIR = 'out'
> DOCKERFILE = "${SOURCEDIR}/dev-support/docker/Dockerfile"
> YETUS='yetus'
> // Branch or tag name. Yetus release tags are 'rel/X.Y.Z'
> YETUS_VERSION='rel/0.13.0'
> /skip-
> # URL for user-side presentation in reports and such
> to our artifacts
> _ _ __
> | ___|_ _(_) |_ _ _ __ ___| |
> | |_ / _` | | | | | | '__/ _ \ |
> | _| (_| | | | |_| | | | __/_|
> |_| \__,_|_|_|\__,_|_| \___(_)
> | Vote |Subsystem | Runtime | Comment
>
> | -1 | yetus | 0m 05s | Unprocessed flag(s): --skip-dir
> {code}
> It seems that the "{{--skip-dir}}" option supported Yetus release prior to
> 0.11.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-17557) skip-dir option is not processed by Yetus
[
https://issues.apache.org/jira/browse/HADOOP-17557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein reassigned HADOOP-17557:
--
Assignee: Ahmed Hussein
> skip-dir option is not processed by Yetus
> -
>
> Key: HADOOP-17557
> URL: https://issues.apache.org/jira/browse/HADOOP-17557
> Project: Hadoop Common
> Issue Type: Bug
> Components: build, precommit, yetus
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
>
> Running test patch locally does not work anymore after the Yetus upgrade
> {code:bash}
> dev-support/bin/test-patch --plugins="maven,checkstyle" --test-parallel=true
> patch-file.patch
> {code}
> Error is
> {code:bash}
> Testing patch on trunk.
> ERROR: Unprocessed flag(s): --skip-dir
> environment {
> SOURCEDIR = 'src'
> // will also need to change notification section below
> PATCHDIR = 'out'
> DOCKERFILE = "${SOURCEDIR}/dev-support/docker/Dockerfile"
> YETUS='yetus'
> // Branch or tag name. Yetus release tags are 'rel/X.Y.Z'
> YETUS_VERSION='rel/0.13.0'
> /skip-
> # URL for user-side presentation in reports and such
> to our artifacts
> _ _ __
> | ___|_ _(_) |_ _ _ __ ___| |
> | |_ / _` | | | | | | '__/ _ \ |
> | _| (_| | | | |_| | | | __/_|
> |_| \__,_|_|_|\__,_|_| \___(_)
> | Vote |Subsystem | Runtime | Comment
>
> | -1 | yetus | 0m 05s | Unprocessed flag(s): --skip-dir
> {code}
> It seems that the "{{--skip-dir}}" option supported Yetus release prior to
> 0.11.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-17557) skip-dir option is not processed by Yetus
[
https://issues.apache.org/jira/browse/HADOOP-17557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HADOOP-17557 started by Ahmed Hussein.
--
> skip-dir option is not processed by Yetus
> -
>
> Key: HADOOP-17557
> URL: https://issues.apache.org/jira/browse/HADOOP-17557
> Project: Hadoop Common
> Issue Type: Bug
> Components: build, precommit, yetus
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
>
> Running test patch locally does not work anymore after the Yetus upgrade
> {code:bash}
> dev-support/bin/test-patch --plugins="maven,checkstyle" --test-parallel=true
> patch-file.patch
> {code}
> Error is
> {code:bash}
> Testing patch on trunk.
> ERROR: Unprocessed flag(s): --skip-dir
> environment {
> SOURCEDIR = 'src'
> // will also need to change notification section below
> PATCHDIR = 'out'
> DOCKERFILE = "${SOURCEDIR}/dev-support/docker/Dockerfile"
> YETUS='yetus'
> // Branch or tag name. Yetus release tags are 'rel/X.Y.Z'
> YETUS_VERSION='rel/0.13.0'
> /skip-
> # URL for user-side presentation in reports and such
> to our artifacts
> _ _ __
> | ___|_ _(_) |_ _ _ __ ___| |
> | |_ / _` | | | | | | '__/ _ \ |
> | _| (_| | | | |_| | | | __/_|
> |_| \__,_|_|_|\__,_|_| \___(_)
> | Vote |Subsystem | Runtime | Comment
>
> | -1 | yetus | 0m 05s | Unprocessed flag(s): --skip-dir
> {code}
> It seems that the "{{--skip-dir}}" option supported Yetus release prior to
> 0.11.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17557) skip-dirs option is not processed by Yetus
[
https://issues.apache.org/jira/browse/HADOOP-17557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated HADOOP-17557:
---
Description:
Running test patch locally does not work anymore after the Yetus upgrade
{code:bash}
dev-support/bin/test-patch --plugins="maven,checkstyle" --test-parallel=true
patch-file.patch
{code}
Error is
{code:bash}
Testing patch on trunk.
ERROR: Unprocessed flag(s): --skip-dir
environment {
SOURCEDIR = 'src'
// will also need to change notification section below
PATCHDIR = 'out'
DOCKERFILE = "${SOURCEDIR}/dev-support/docker/Dockerfile"
YETUS='yetus'
// Branch or tag name. Yetus release tags are 'rel/X.Y.Z'
YETUS_VERSION='rel/0.13.0'
/skip-
# URL for user-side presentation in reports and such to
our artifacts
_ _ __
| ___|_ _(_) |_ _ _ __ ___| |
| |_ / _` | | | | | | '__/ _ \ |
| _| (_| | | | |_| | | | __/_|
|_| \__,_|_|_|\__,_|_| \___(_)
| Vote |Subsystem | Runtime | Comment
| -1 | yetus | 0m 05s | Unprocessed flag(s): --skip-dir
{code}
It seems that the "{{--skip-dir}}" option supported Yetus release prior to 0.11.
was:
Running test patch locally does not work anymore after the Yetus upgrade
{code:bash}
dev-support/bin/test-patch --plugins="maven,checkstyle" --test-parallel=true
patch-file.patch
{code}
Error is
{code:bash}
Testing patch on trunk.
ERROR: Unprocessed flag(s): --skip-dir
environment {
SOURCEDIR = 'src'
// will also need to change notification section below
PATCHDIR = 'out'
DOCKERFILE = "${SOURCEDIR}/dev-support/docker/Dockerfile"
YETUS='yetus'
// Branch or tag name. Yetus release tags are 'rel/X.Y.Z'
YETUS_VERSION='rel/0.13.0'
/skip-
# URL for user-side presentation in reports and such to
our artifacts
_ _ __
| ___|_ _(_) |_ _ _ __ ___| |
| |_ / _` | | | | | | '__/ _ \ |
| _| (_| | | | |_| | | | __/_|
|_| \__,_|_|_|\__,_|_| \___(_)
| Vote |Subsystem | Runtime | Comment
| -1 | yetus | 0m 05s | Unprocessed flag(s): --skip-dir
{code}
It seems that the "{{--skip-dir}}" option was never supported by any Yetus
release.
> skip-dirs option is not processed by Yetus
> --
>
> Key: HADOOP-17557
> URL: https://issues.apache.org/jira/browse/HADOOP-17557
> Project: Hadoop Common
> Issue Type: Bug
> Components: build, precommit, yetus
>Reporter: Ahmed Hussein
>Priority: Major
>
> Running test patch locally does not work anymore after the Yetus upgrade
> {code:bash}
> dev-support/bin/test-patch --plugins="maven,checkstyle" --test-parallel=true
> patch-file.patch
> {code}
> Error is
> {code:bash}
> Testing patch on trunk.
> ERROR: Unprocessed flag(s): --skip-dir
> environment {
> SOURCEDIR = 'src'
> // will also need to change notification section below
> PATCHDIR = 'out'
> DOCKERFILE = "${SOURCEDIR}/dev-support/docker/Dockerfile"
> YETUS='yetus'
> // Branch or tag name. Yetus release tags are 'rel/X.Y.Z'
> YETUS_VERSION='rel/0.13.0'
> /skip-
> # URL for user-side presentation in reports and such
> to our artifacts
> _ _ __
> | ___|_ _(_) |_ _ _ __ ___| |
> | |_ / _` | | | | | | '__/ _ \ |
> | _| (_| | | | |_| | | | __/_|
> |_| \__,_|_|_|\__,_|_| \___(_)
> | Vote |Subsystem | Runtime | Comment
>
> | -1 | yetus | 0m 05s | Unprocessed flag(s): --skip-dir
> {code}
> It seems that the "{{--skip-dir}}" option supported Yetus release prior to
> 0.11.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17557) skip-dir option is not processed by Yetus
[
https://issues.apache.org/jira/browse/HADOOP-17557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated HADOOP-17557:
---
Summary: skip-dir option is not processed by Yetus (was: skip-dirs option
is not processed by Yetus)
> skip-dir option is not processed by Yetus
> -
>
> Key: HADOOP-17557
> URL: https://issues.apache.org/jira/browse/HADOOP-17557
> Project: Hadoop Common
> Issue Type: Bug
> Components: build, precommit, yetus
>Reporter: Ahmed Hussein
>Priority: Major
>
> Running test patch locally does not work anymore after the Yetus upgrade
> {code:bash}
> dev-support/bin/test-patch --plugins="maven,checkstyle" --test-parallel=true
> patch-file.patch
> {code}
> Error is
> {code:bash}
> Testing patch on trunk.
> ERROR: Unprocessed flag(s): --skip-dir
> environment {
> SOURCEDIR = 'src'
> // will also need to change notification section below
> PATCHDIR = 'out'
> DOCKERFILE = "${SOURCEDIR}/dev-support/docker/Dockerfile"
> YETUS='yetus'
> // Branch or tag name. Yetus release tags are 'rel/X.Y.Z'
> YETUS_VERSION='rel/0.13.0'
> /skip-
> # URL for user-side presentation in reports and such
> to our artifacts
> _ _ __
> | ___|_ _(_) |_ _ _ __ ___| |
> | |_ / _` | | | | | | '__/ _ \ |
> | _| (_| | | | |_| | | | __/_|
> |_| \__,_|_|_|\__,_|_| \___(_)
> | Vote |Subsystem | Runtime | Comment
>
> | -1 | yetus | 0m 05s | Unprocessed flag(s): --skip-dir
> {code}
> It seems that the "{{--skip-dir}}" option supported Yetus release prior to
> 0.11.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] dineshchitlangia merged pull request #2743: HDFS-15873. Add namenode address in logs for block report
dineshchitlangia merged pull request #2743: URL: https://github.com/apache/hadoop/pull/2743 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #2741: HDFS-15855.Solve the problem of incorrect EC progress when loading FsImage.
goiri commented on a change in pull request #2741:
URL: https://github.com/apache/hadoop/pull/2741#discussion_r589533032
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEnabledECPolicies.java
##
@@ -143,7 +143,7 @@ public void testChangeDefaultPolicy() throws Exception {
testPolicy);
manager.init(conf);
// Load policies similar to when fsimage is loaded at namenode startup
-manager.loadPolicies(constructAllDisabledInitialPolicies(), conf);
+manager.loadPolicies(constructAllDisabledInitialPolicies(), conf, null);
Review comment:
Can we add a test with the counter being passed?
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingPolicyManager.java
##
@@ -482,11 +483,15 @@ private void loadPolicy(ErasureCodingPolicyInfo info) {
*
*/
public synchronized void loadPolicies(
- List ecPolicies, Configuration conf)
+ List ecPolicies, Configuration conf,
Review comment:
Can you leave the old header passing a null so we don't need to modify
the tests?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on pull request #2743: HDFS-15873. Add namenode address in logs for block report
tomscut commented on pull request #2743: URL: https://github.com/apache/hadoop/pull/2743#issuecomment-792822951 Hi @dineshchitlangia , could you please help review the code? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #2746: HDFS-15875. Check whether file is being truncated before truncate
ayushtkn commented on a change in pull request #2746:
URL: https://github.com/apache/hadoop/pull/2746#discussion_r589419978
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
##
@@ -631,10 +698,10 @@ public void testTruncateFailure() throws IOException {
{
try {
fs.truncate(p, 0);
-fail("Truncate must fail since a trancate is already in pregress.");
+fail("Truncate must fail since a truncate is already in progress.");
} catch (IOException expected) {
GenericTestUtils.assertExceptionContains(
-"Failed to TRUNCATE_FILE", expected);
+"/dir/testTruncateFailure is being truncated", expected);
}
Review comment:
What is the difference between this case and ours? Why doesn't this
exception triggers for our case? and can we accommodate our fix to the check
throwing this exception
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17531) DistCp: Reduce memory usage on copying huge directories
[
https://issues.apache.org/jira/browse/HADOOP-17531?focusedWorklogId=562320=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-562320
]
ASF GitHub Bot logged work on HADOOP-17531:
---
Author: ASF GitHub Bot
Created on: 08/Mar/21 12:13
Start Date: 08/Mar/21 12:13
Worklog Time Spent: 10m
Work Description: jojochuang commented on a change in pull request #2732:
URL: https://github.com/apache/hadoop/pull/2732#discussion_r589370003
##
File path:
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java
##
@@ -688,6 +696,71 @@ private void traverseDirectory(SequenceFile.Writer
fileListWriter,
workers.shutdown();
}
+ private void traverseDirectoryLegacy(SequenceFile.Writer fileListWriter,
+ FileSystem sourceFS, ArrayList sourceDirs,
+ Path sourcePathRoot, DistCpContext context, HashSet excludeList,
+ List fileStatuses) throws IOException {
+final boolean preserveAcls = context.shouldPreserve(FileAttribute.ACL);
+final boolean preserveXAttrs = context.shouldPreserve(FileAttribute.XATTR);
+final boolean preserveRawXattrs = context.shouldPreserveRawXattrs();
+
+Stack pathStack = new Stack();
+for (FileStatus fs : sourceDirs) {
+ if (excludeList == null || !excludeList
+ .contains(fs.getPath().toUri().getPath())) {
+pathStack.add(fs);
+ }
+}
+
+while (!pathStack.isEmpty()) {
+ prepareListing(pathStack.pop().getPath(), fileListWriter, sourceFS,
+ sourcePathRoot, context, excludeList, fileStatuses, preserveAcls,
+ preserveXAttrs, preserveRawXattrs);
+}
+ }
+
+ @SuppressWarnings("checkstyle:parameternumber")
+ private void prepareListing(Path path, SequenceFile.Writer fileListWriter,
Review comment:
Sorry for coming back late.
Can we refactor this method a bit to use fewer parameters?
##
File path: hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm
##
@@ -362,6 +362,7 @@ Command Line Options
| `-copybuffersize ` | Size of the copy buffer to use. By
default, `` is set to 8192B | |
| `-xtrack ` | Save information about missing source files to the
specified path. | This option is only valid with `-update` option. This is an
experimental property and it cannot be used with `-atomic` option. |
| `-direct` | Write directly to destination paths | Useful for avoiding
potentially very expensive temporary file rename operations when the
destination is an object store |
+| `-useIterator` | Uses single threaded listStatusIterator to build listing |
Useful for saving memory at the client side. |
Review comment:
I was think we shouuld update the doc to mention it will disable
-numListstatusThreads. But if we can merge that WIP patch soon then it's fine.
##
File path:
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java
##
@@ -688,6 +696,71 @@ private void traverseDirectory(SequenceFile.Writer
fileListWriter,
workers.shutdown();
}
+ private void traverseDirectoryLegacy(SequenceFile.Writer fileListWriter,
+ FileSystem sourceFS, ArrayList sourceDirs,
+ Path sourcePathRoot, DistCpContext context, HashSet excludeList,
+ List fileStatuses) throws IOException {
+final boolean preserveAcls = context.shouldPreserve(FileAttribute.ACL);
+final boolean preserveXAttrs = context.shouldPreserve(FileAttribute.XATTR);
+final boolean preserveRawXattrs = context.shouldPreserveRawXattrs();
+
+Stack pathStack = new Stack();
+for (FileStatus fs : sourceDirs) {
+ if (excludeList == null || !excludeList
+ .contains(fs.getPath().toUri().getPath())) {
+pathStack.add(fs);
+ }
+}
+
+while (!pathStack.isEmpty()) {
+ prepareListing(pathStack.pop().getPath(), fileListWriter, sourceFS,
+ sourcePathRoot, context, excludeList, fileStatuses, preserveAcls,
+ preserveXAttrs, preserveRawXattrs);
+}
+ }
+
+ @SuppressWarnings("checkstyle:parameternumber")
+ private void prepareListing(Path path, SequenceFile.Writer fileListWriter,
Review comment:
maybe we should refactor traverseDirectory() into a class since we pass
over the parameters here and there.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 562320)
Time Spent: 50m (was: 40m)
> DistCp: Reduce memory usage on copying huge directories
> ---
>
> Key: HADOOP-17531
> URL:
[GitHub] [hadoop] jojochuang commented on a change in pull request #2732: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories.
jojochuang commented on a change in pull request #2732:
URL: https://github.com/apache/hadoop/pull/2732#discussion_r589370003
##
File path:
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java
##
@@ -688,6 +696,71 @@ private void traverseDirectory(SequenceFile.Writer
fileListWriter,
workers.shutdown();
}
+ private void traverseDirectoryLegacy(SequenceFile.Writer fileListWriter,
+ FileSystem sourceFS, ArrayList sourceDirs,
+ Path sourcePathRoot, DistCpContext context, HashSet excludeList,
+ List fileStatuses) throws IOException {
+final boolean preserveAcls = context.shouldPreserve(FileAttribute.ACL);
+final boolean preserveXAttrs = context.shouldPreserve(FileAttribute.XATTR);
+final boolean preserveRawXattrs = context.shouldPreserveRawXattrs();
+
+Stack pathStack = new Stack();
+for (FileStatus fs : sourceDirs) {
+ if (excludeList == null || !excludeList
+ .contains(fs.getPath().toUri().getPath())) {
+pathStack.add(fs);
+ }
+}
+
+while (!pathStack.isEmpty()) {
+ prepareListing(pathStack.pop().getPath(), fileListWriter, sourceFS,
+ sourcePathRoot, context, excludeList, fileStatuses, preserveAcls,
+ preserveXAttrs, preserveRawXattrs);
+}
+ }
+
+ @SuppressWarnings("checkstyle:parameternumber")
+ private void prepareListing(Path path, SequenceFile.Writer fileListWriter,
Review comment:
Sorry for coming back late.
Can we refactor this method a bit to use fewer parameters?
##
File path: hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm
##
@@ -362,6 +362,7 @@ Command Line Options
| `-copybuffersize ` | Size of the copy buffer to use. By
default, `` is set to 8192B | |
| `-xtrack ` | Save information about missing source files to the
specified path. | This option is only valid with `-update` option. This is an
experimental property and it cannot be used with `-atomic` option. |
| `-direct` | Write directly to destination paths | Useful for avoiding
potentially very expensive temporary file rename operations when the
destination is an object store |
+| `-useIterator` | Uses single threaded listStatusIterator to build listing |
Useful for saving memory at the client side. |
Review comment:
I was think we shouuld update the doc to mention it will disable
-numListstatusThreads. But if we can merge that WIP patch soon then it's fine.
##
File path:
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java
##
@@ -688,6 +696,71 @@ private void traverseDirectory(SequenceFile.Writer
fileListWriter,
workers.shutdown();
}
+ private void traverseDirectoryLegacy(SequenceFile.Writer fileListWriter,
+ FileSystem sourceFS, ArrayList sourceDirs,
+ Path sourcePathRoot, DistCpContext context, HashSet excludeList,
+ List fileStatuses) throws IOException {
+final boolean preserveAcls = context.shouldPreserve(FileAttribute.ACL);
+final boolean preserveXAttrs = context.shouldPreserve(FileAttribute.XATTR);
+final boolean preserveRawXattrs = context.shouldPreserveRawXattrs();
+
+Stack pathStack = new Stack();
+for (FileStatus fs : sourceDirs) {
+ if (excludeList == null || !excludeList
+ .contains(fs.getPath().toUri().getPath())) {
+pathStack.add(fs);
+ }
+}
+
+while (!pathStack.isEmpty()) {
+ prepareListing(pathStack.pop().getPath(), fileListWriter, sourceFS,
+ sourcePathRoot, context, excludeList, fileStatuses, preserveAcls,
+ preserveXAttrs, preserveRawXattrs);
+}
+ }
+
+ @SuppressWarnings("checkstyle:parameternumber")
+ private void prepareListing(Path path, SequenceFile.Writer fileListWriter,
Review comment:
maybe we should refactor traverseDirectory() into a class since we pass
over the parameters here and there.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-17531) DistCp: Reduce memory usage on copying huge directories
[ https://issues.apache.org/jira/browse/HADOOP-17531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena reassigned HADOOP-17531: - Assignee: Ayush Saxena > DistCp: Reduce memory usage on copying huge directories > --- > > Key: HADOOP-17531 > URL: https://issues.apache.org/jira/browse/HADOOP-17531 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Critical > Labels: pull-request-available > Attachments: MoveToStackIterator.patch, gc-NewD-512M-3.8ML.log > > Time Spent: 40m > Remaining Estimate: 0h > > Presently distCp, uses the producer-consumer kind of setup while building the > listing, the input queue and output queue are both unbounded, thus the > listStatus grows quite huge. > Rel Code Part : > https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java#L635 > This goes on bredth-first traversal kind of stuff(uses queue instead of > earlier stack), so if you have files at lower depth, it will like open up the > entire tree and the start processing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17559) S3Guard import can OOM on large imports
[ https://issues.apache.org/jira/browse/HADOOP-17559?focusedWorklogId=562303=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-562303 ] ASF GitHub Bot logged work on HADOOP-17559: --- Author: ASF GitHub Bot Created on: 08/Mar/21 11:06 Start Date: 08/Mar/21 11:06 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2734: URL: https://github.com/apache/hadoop/pull/2734#issuecomment-792678632 +Add a test to verify we can import with spaces in the filenames This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 562303) Time Spent: 40m (was: 0.5h) > S3Guard import can OOM on large imports > --- > > Key: HADOOP-17559 > URL: https://issues.apache.org/jira/browse/HADOOP-17559 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > I know I'm closing ~all S3Guard issues as wontfix, but this is pressing so > I'm going to do it anyway > S3guard import of directory tree containing many, many files will OOM. > Looking at the code this is going to be because > * import tool builds a map of all dirs imported, which as the comments note > "superfluous for DDB". - *cut* > * DDB AncestorState tracks files as well as dirs, purely as a safety check to > make sure current op doesn't somehow write a file entry above a dir entry in > the same operation > We've been running S3Guard for a long time, and condition #2 has never arisen. > Propose: don't store filenames there, so memory consumption goes from O(files > + dirs) to O(dirs) > Code straightforward, can't think of any tests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2734: HADOOP-17559. S3guard import OOM.
steveloughran commented on pull request #2734: URL: https://github.com/apache/hadoop/pull/2734#issuecomment-792678632 +Add a test to verify we can import with spaces in the filenames This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17568) Mapred/YARN job fails due to kms-dt can't be found in cache with LoadBalancingKMSClientProvider + Kerberos
[
https://issues.apache.org/jira/browse/HADOOP-17568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zbigniew Kostrzewa updated HADOOP-17568:
Description:
I deployed Hadoop 3.2.2 cluster with KMS in HA using
LoadBalancingKMSClientProvider with Kerberos authentication. KMS instances are
configured with ZooKeeper for storing the shared secret.
I have created an encryption key and an encryption zone in `/test` directory
and executed `randomtextwriter` from mapreduce examples passing it a
sub-directory in the encryption zone:
{code:java}
hadoop jar hadoop-mapreduce-examples-3.2.2.jar randomtextwriter
/test/randomtextwriter
{code}
Unfortunately the job keeps failing with errors like:
{code:java}
java.io.IOException:
org.apache.hadoop.security.authentication.client.AuthenticationException:
org.apache.hadoop.security.token.SecretManager$InvalidToken: token (kms-dt
owner=packer, renewer=packer, realUser=, issueDate=1615146155993,
maxDate=1615750955993, sequenceNumber=1, masterKeyId=2) can't be found in cache
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:363)
at
org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:532)
at
org.apache.hadoop.hdfs.HdfsKMSUtil.decryptEncryptedDataEncryptionKey(HdfsKMSUtil.java:212)
at
org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:972)
at
org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:952)
at
org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:536)
at
org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:530)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:544)
at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:471)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1125)
at
org.apache.hadoop.io.SequenceFile$Writer.(SequenceFile.java:1168)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:285)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:542)
at
org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:64)
at
org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75)
at
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.(MapTask.java:659)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:779)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by:
org.apache.hadoop.security.authentication.client.AuthenticationException:
org.apache.hadoop.security.token.SecretManager$InvalidToken: token (kms-dt
owner=packer, renewer=packer, realUser=, issueDate=1615146155993,
maxDate=1615750955993, sequenceNumber=1, masterKeyId=2) can't be found in cache
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:154)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:592)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:540)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:833)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:356)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:352)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:174)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:352)
{code}
I've injected a few logs on my own and it seems that the client gets 403 on
"decrypt"
[GitHub] [hadoop] LeonGao91 commented on a change in pull request #2738: HDFS-15842. HDFS mover to emit metrics.
LeonGao91 commented on a change in pull request #2738: URL: https://github.com/apache/hadoop/pull/2738#discussion_r589234446 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/package-info.java ## @@ -0,0 +1,27 @@ +/** Review comment: It is to pass the checkstyle as I added a new file to this package without package-info.. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] LeonGao91 commented on a change in pull request #2738: HDFS-15842. HDFS mover to emit metrics.
LeonGao91 commented on a change in pull request #2738:
URL: https://github.com/apache/hadoop/pull/2738#discussion_r589230766
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java
##
@@ -649,6 +664,11 @@ static int run(Map> namenodes,
Configuration conf)
Map> excludedPinnedBlocks = new HashMap<>();
LOG.info("namenodes = " + namenodes);
+DefaultMetricsSystem.initialize("Mover");
Review comment:
We have it in the internal branch with similar implementation but it is
not in upstream yet.
Actually, there has been a JIRA long time ago for balancer metrics.
https://issues.apache.org/jira/browse/HDFS-10648
I can rebase our changes there as a different PR after this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17568) Mapred/YARN job fails due to kms-dt can't be found in cache with LoadBalancingKMSClientProvider + Kerberos
[
https://issues.apache.org/jira/browse/HADOOP-17568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17297151#comment-17297151
]
Akira Ajisaka commented on HADOOP-17568:
Yes, now I use both LoadBalancingKMSClientProvider and VIP.
> Mapred/YARN job fails due to kms-dt can't be found in cache with
> LoadBalancingKMSClientProvider + Kerberos
> --
>
> Key: HADOOP-17568
> URL: https://issues.apache.org/jira/browse/HADOOP-17568
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms, security
>Affects Versions: 3.2.2
>Reporter: Zbigniew Kostrzewa
>Priority: Major
>
> I deployed Hadoop 3.2.2 cluster with KMS in HA using
> LoadBalancingKMSClientProvider with Kerberos authentication. KMS instances
> are configured with ZooKeeper for storing the shared secret.
> I have created an encryption key and an encryption zone in `/test` directory
> and executed `randomtextwriter` from mapreduce examples passing it a
> sub-directory in the encryption zone:
> {code:java}
> hadoop jar hadoop-mapreduce-examples-3.2.2.jar randomtextwriter
> /test/randomtextwriter
> {code}
> Unfortunately the job keeps failing with errors like:
> {code:java}
> java.io.IOException:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: token (kms-dt
> owner=packer, renewer=packer, realUser=, issueDate=1615146155993,
> maxDate=1615750955993, sequenceNumber=1, masterKeyId=2) can't be found in
> cache
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:363)
> at
> org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:532)
> at
> org.apache.hadoop.hdfs.HdfsKMSUtil.decryptEncryptedDataEncryptionKey(HdfsKMSUtil.java:212)
> at
> org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:972)
> at
> org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:952)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:536)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:530)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:544)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:471)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1125)
> at
> org.apache.hadoop.io.SequenceFile$Writer.(SequenceFile.java:1168)
> at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:285)
> at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:542)
> at
> org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:64)
> at
> org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75)
> at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.(MapTask.java:659)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:779)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> Caused by:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: token (kms-dt
> owner=packer, renewer=packer, realUser=, issueDate=1615146155993,
> maxDate=1615750955993, sequenceNumber=1, masterKeyId=2) can't be found in
> cache
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:154)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:592)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:540)
> at
>
[jira] [Commented] (HADOOP-17568) Mapred/YARN job fails due to kms-dt can't be found in cache with LoadBalancingKMSClientProvider + Kerberos
[
https://issues.apache.org/jira/browse/HADOOP-17568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17297149#comment-17297149
]
Zbigniew Kostrzewa commented on HADOOP-17568:
-
[~aajisaka]
Thanks your comment. Actually I had
`hadoop.kms.authentication.signer.secret.provider.zookeeper.auth.type` set to
`none`. Nevertheless, I just re-configured my ZooKeeper to use Kerberos and
updated `kms-site.xml` accordingly. Unfortunately, that did not help.
I added to the description contents of `kms-site.xml` from one of my KMS
instances (without Kerberos for ZooKeeper).
Do you use *LoadBalancingKMSClientProvider* or *Load-Balancer or VIP*?
> Mapred/YARN job fails due to kms-dt can't be found in cache with
> LoadBalancingKMSClientProvider + Kerberos
> --
>
> Key: HADOOP-17568
> URL: https://issues.apache.org/jira/browse/HADOOP-17568
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms, security
>Affects Versions: 3.2.2
>Reporter: Zbigniew Kostrzewa
>Priority: Major
>
> I deployed Hadoop 3.2.2 cluster with KMS in HA using
> LoadBalancingKMSClientProvider with Kerberos authentication. KMS instances
> are configured with ZooKeeper for storing the shared secret.
> I have created an encryption key and an encryption zone in `/test` directory
> and executed `randomtextwriter` from mapreduce examples passing it a
> sub-directory in the encryption zone:
> {code:java}
> hadoop jar hadoop-mapreduce-examples-3.2.2.jar randomtextwriter
> /test/randomtextwriter
> {code}
> Unfortunately the job keeps failing with errors like:
> {code:java}
> java.io.IOException:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: token (kms-dt
> owner=packer, renewer=packer, realUser=, issueDate=1615146155993,
> maxDate=1615750955993, sequenceNumber=1, masterKeyId=2) can't be found in
> cache
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:363)
> at
> org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:532)
> at
> org.apache.hadoop.hdfs.HdfsKMSUtil.decryptEncryptedDataEncryptionKey(HdfsKMSUtil.java:212)
> at
> org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:972)
> at
> org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:952)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:536)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:530)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:544)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:471)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1125)
> at
> org.apache.hadoop.io.SequenceFile$Writer.(SequenceFile.java:1168)
> at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:285)
> at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:542)
> at
> org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:64)
> at
> org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75)
> at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.(MapTask.java:659)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:779)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> Caused by:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: token (kms-dt
> owner=packer, renewer=packer, realUser=, issueDate=1615146155993,
> maxDate=1615750955993, sequenceNumber=1, masterKeyId=2) can't be found in
> cache
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
>
[GitHub] [hadoop] LeonGao91 commented on a change in pull request #2738: HDFS-15842. HDFS mover to emit metrics.
LeonGao91 commented on a change in pull request #2738:
URL: https://github.com/apache/hadoop/pull/2738#discussion_r589230766
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java
##
@@ -649,6 +664,11 @@ static int run(Map> namenodes,
Configuration conf)
Map> excludedPinnedBlocks = new HashMap<>();
LOG.info("namenodes = " + namenodes);
+DefaultMetricsSystem.initialize("Mover");
Review comment:
We have it in the internal branch with similar implementation but it is
not in upstream.
Surely, we can put the changes together. Let me update the JIRA to include
the balancer metrics.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17568) Mapred/YARN job fails due to kms-dt can't be found in cache with LoadBalancingKMSClientProvider + Kerberos
[
https://issues.apache.org/jira/browse/HADOOP-17568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zbigniew Kostrzewa updated HADOOP-17568:
Description:
I deployed Hadoop 3.2.2 cluster with KMS in HA using
LoadBalancingKMSClientProvider with Kerberos authentication. KMS instances are
configured with ZooKeeper for storing the shared secret.
I have created an encryption key and an encryption zone in `/test` directory
and executed `randomtextwriter` from mapreduce examples passing it a
sub-directory in the encryption zone:
{code:java}
hadoop jar hadoop-mapreduce-examples-3.2.2.jar randomtextwriter
/test/randomtextwriter
{code}
Unfortunately the job keeps failing with errors like:
{code:java}
java.io.IOException:
org.apache.hadoop.security.authentication.client.AuthenticationException:
org.apache.hadoop.security.token.SecretManager$InvalidToken: token (kms-dt
owner=packer, renewer=packer, realUser=, issueDate=1615146155993,
maxDate=1615750955993, sequenceNumber=1, masterKeyId=2) can't be found in cache
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:363)
at
org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:532)
at
org.apache.hadoop.hdfs.HdfsKMSUtil.decryptEncryptedDataEncryptionKey(HdfsKMSUtil.java:212)
at
org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:972)
at
org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:952)
at
org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:536)
at
org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:530)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:544)
at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:471)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1125)
at
org.apache.hadoop.io.SequenceFile$Writer.(SequenceFile.java:1168)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:285)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:542)
at
org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:64)
at
org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75)
at
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.(MapTask.java:659)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:779)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by:
org.apache.hadoop.security.authentication.client.AuthenticationException:
org.apache.hadoop.security.token.SecretManager$InvalidToken: token (kms-dt
owner=packer, renewer=packer, realUser=, issueDate=1615146155993,
maxDate=1615750955993, sequenceNumber=1, masterKeyId=2) can't be found in cache
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:154)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:592)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:540)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:833)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:356)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:352)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:174)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:352)
{code}
I've injected a few logs on my own and it seems that the client gets 403 on
"decrypt"
