[GitHub] [hadoop] hadoop-yetus commented on issue #1341: HDDS-2022. Add additional freon tests
hadoop-yetus commented on issue #1341: HDDS-2022. Add additional freon tests URL: https://github.com/apache/hadoop/pull/1341#issuecomment-531452566 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 153 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | -1 | mvninstall | 35 | hadoop-ozone in trunk failed. | | -1 | compile | 22 | hadoop-ozone in trunk failed. | | -0 | checkstyle | 37 | The patch fails to run checkstyle in hadoop-ozone | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 990 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 100 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 187 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 28 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | -1 | mvninstall | 35 | hadoop-ozone in the patch failed. | | -1 | compile | 25 | hadoop-ozone in the patch failed. | | -1 | javac | 25 | hadoop-ozone in the patch failed. | | -0 | checkstyle | 38 | The patch fails to run checkstyle in hadoop-ozone | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 1 | The patch has no ill-formed XML file. | | +1 | shadedclient | 821 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 91 | hadoop-ozone in the patch failed. | | -1 | findbugs | 28 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 198 | hadoop-hdds in the patch failed. | | -1 | unit | 30 | hadoop-ozone in the patch failed. | | +1 | asflicense | 34 | The patch does not generate ASF License warnings. | | | | 3684 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.ozoneimpl.TestOzoneContainer | | | hadoop.ozone.container.keyvalue.TestKeyValueContainer | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1341/10/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1341 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle | | uname | Linux e83069eba372 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 9a931b8 | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1341/10/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1341/10/artifact/out/branch-compile-hadoop-ozone.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1341/10/artifact/out//home/jenkins/jenkins-slave/workspace/hadoop-multibranch_PR-1341/out/maven-branch-checkstyle-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1341/10/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1341/10/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1341/10/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1341/10/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1341/10/artifact/out/patch-compile-hadoop-ozone.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1341/10/artifact/out//home/jenkins/jenkins-slave/workspace/hadoop-multibranch_PR-1341/out/maven-patch-checkstyle-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1341/10/artifact/out/patch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1341/10/artifact/out/patch-findbugs-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1341/10/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1341/10/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results |
[GitHub] [hadoop] elek commented on issue #1435: HDDS-2119. Use checkstyle.xml and suppressions.xml in hdds/ozone projects for checkstyle validation.
elek commented on issue #1435: HDDS-2119. Use checkstyle.xml and suppressions.xml in hdds/ozone projects for checkstyle validation. URL: https://github.com/apache/hadoop/pull/1435#issuecomment-531450551 Thanks @nandakumar131 the patch. Unfortunately it seems to be failing for me with testing it with the checkstyle.sh. Until now the checkstyle.sh did a `mvn checkstyle:check` without compilation (which made it very fast). But now you should have the build-tools compiled and installed to do a full checkstyle check. I don't know what is the good solution here 1. We can use file references instead of artifact references to define the checkstyle rule files 2. Or we can add the required build-tools install to the checkstyle.sh I liked that it was very fast until know, therefore I would prefer the first one, but there could be other options... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on a change in pull request #1344: HDDS-1982 Extend SCMNodeManager to support decommission and maintenance states
anuengineer commented on a change in pull request #1344: HDDS-1982 Extend
SCMNodeManager to support decommission and maintenance states
URL: https://github.com/apache/hadoop/pull/1344#discussion_r324412721
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/states/NodeStateMap.java
##
@@ -309,4 +381,61 @@ private void checkIfNodeExist(UUID uuid) throws
NodeNotFoundException {
throw new NodeNotFoundException("Node UUID: " + uuid);
}
}
+
+ /**
+ * Create a list of datanodeInfo for all nodes matching the passed states.
+ * Passing null for one of the states acts like a wildcard for that state.
+ *
+ * @param opState
+ * @param health
+ * @return List of DatanodeInfo objects matching the passed state
+ */
+ private List filterNodes(
+ NodeOperationalState opState, NodeState health) {
+if (opState != null && health != null) {
Review comment:
please be aware that stream.filter kind of patterns have a huge overhead
over normal for. If this code is going to be in any sort of critical path, it
is better for the code to stay normal for.
Please see some fixes made by todd lipcon, in HDFS because of this issue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1448: HDDS-2110. Arbitrary file can be downloaded with the help of ProfilerServlet
hadoop-yetus commented on issue #1448: HDDS-2110. Arbitrary file can be downloaded with the help of ProfilerServlet URL: https://github.com/apache/hadoop/pull/1448#issuecomment-531449913 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 72 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | -1 | mvninstall | 31 | hadoop-ozone in trunk failed. | | -1 | compile | 19 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 64 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 902 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 14 | hadoop-hdds in trunk failed. | | -1 | javadoc | 15 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 160 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 23 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | -1 | mvninstall | 30 | hadoop-ozone in the patch failed. | | -1 | compile | 22 | hadoop-ozone in the patch failed. | | -1 | javac | 22 | hadoop-ozone in the patch failed. | | +1 | checkstyle | 51 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 1 | The patch has no whitespace issues. | | +1 | shadedclient | 721 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 13 | hadoop-hdds in the patch failed. | | -1 | javadoc | 14 | hadoop-ozone in the patch failed. | | -1 | findbugs | 164 | hadoop-hdds generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | -1 | findbugs | 22 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 152 | hadoop-hdds in the patch failed. | | -1 | unit | 24 | hadoop-ozone in the patch failed. | | +1 | asflicense | 29 | The patch does not generate ASF License warnings. | | | | 2927 | | | Reason | Tests | |---:|:--| | FindBugs | module:hadoop-hdds | | | Absolute path traversal in org.apache.hadoop.hdds.server.ProfileServlet.doGet(HttpServletRequest, HttpServletResponse) At ProfileServlet.java:HttpServletResponse) At ProfileServlet.java:[line 181] | | Failed junit tests | hadoop.ozone.container.ozoneimpl.TestOzoneContainer | | | hadoop.ozone.container.keyvalue.TestKeyValueContainer | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1448/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1448 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux c3afa8e01794 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 6a9f7ca | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1448/1/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1448/1/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1448/1/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1448/1/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1448/1/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1448/1/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1448/1/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1448/1/artifact/out/patch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1448/1/artifact/out/patch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1448/1/artifact/out/patch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1448/1/artifact/out/new-findbugs-hadoop-hdds.html | | findbugs |
[GitHub] [hadoop] anuengineer commented on issue #1444: HDDS-2078. Get/Renew DelegationToken NPE after HDDS-1909. Contributed…
anuengineer commented on issue #1444: HDDS-2078. Get/Renew DelegationToken NPE after HDDS-1909. Contributed… URL: https://github.com/apache/hadoop/pull/1444#issuecomment-531449080 @xiaoyuyao I think all these failures are due to the trunk issues. If you can retest, it might be a good idea. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on issue #1444: HDDS-2078. Get/Renew DelegationToken NPE after HDDS-1909. Contributed…
anuengineer commented on issue #1444: HDDS-2078. Get/Renew DelegationToken NPE after HDDS-1909. Contributed… URL: https://github.com/apache/hadoop/pull/1444#issuecomment-531449098 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on issue #1448: HDDS-2110. Arbitrary file can be downloaded with the help of ProfilerServlet
anuengineer commented on issue #1448: HDDS-2110. Arbitrary file can be downloaded with the help of ProfilerServlet URL: https://github.com/apache/hadoop/pull/1448#issuecomment-531448967 +1, pending Jenkins. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15804) upgrade to commons-compress 1.18
[ https://issues.apache.org/jira/browse/HADOOP-15804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929680#comment-16929680 ] Wei-Chiu Chuang commented on HADOOP-15804: -- Cherrypicked the commit into branch2. > upgrade to commons-compress 1.18 > > > Key: HADOOP-15804 > URL: https://issues.apache.org/jira/browse/HADOOP-15804 > Project: Hadoop Common > Issue Type: Improvement >Reporter: PJ Fanning >Assignee: Akira Ajisaka >Priority: Major > Fix For: 2.10.0, 3.0.4, 3.3.0, 3.1.2, 3.2.1 > > Attachments: HADOOP-15804.01.patch > > > [https://github.com/apache/commons-compress/blob/master/RELEASE-NOTES.txt] > Some CVEs have been fixed in recent releases > ([https://commons.apache.org/proper/commons-compress/security-reports.html]) > [https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common/3.1.1] > depends on commons-compress 1.4.1 -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16555) Update commons-compress to 1.19
[ https://issues.apache.org/jira/browse/HADOOP-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-16555: - Fix Version/s: 2.10.0 > Update commons-compress to 1.19 > --- > > Key: HADOOP-16555 > URL: https://issues.apache.org/jira/browse/HADOOP-16555 > Project: Hadoop Common > Issue Type: Task >Reporter: Wei-Chiu Chuang >Assignee: YiSheng Lien >Priority: Major > Labels: release-blocker > Fix For: 2.10.0, 3.3.0, 3.2.1, 3.1.3 > > Attachments: HADOOP-16555.branch-3.2.patch > > > We depend on commons-compress 1.18. The 1.19 release just went out. I think > we should update it. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15804) upgrade to commons-compress 1.18
[ https://issues.apache.org/jira/browse/HADOOP-15804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-15804: - Fix Version/s: 2.10.0 > upgrade to commons-compress 1.18 > > > Key: HADOOP-15804 > URL: https://issues.apache.org/jira/browse/HADOOP-15804 > Project: Hadoop Common > Issue Type: Improvement >Reporter: PJ Fanning >Assignee: Akira Ajisaka >Priority: Major > Fix For: 2.10.0, 3.0.4, 3.3.0, 3.1.2, 3.2.1 > > Attachments: HADOOP-15804.01.patch > > > [https://github.com/apache/commons-compress/blob/master/RELEASE-NOTES.txt] > Some CVEs have been fixed in recent releases > ([https://commons.apache.org/proper/commons-compress/security-reports.html]) > [https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common/3.1.1] > depends on commons-compress 1.4.1 -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] elek commented on a change in pull request #1341: HDDS-2022. Add additional freon tests
elek commented on a change in pull request #1341: HDDS-2022. Add additional
freon tests
URL: https://github.com/apache/hadoop/pull/1341#discussion_r324412003
##
File path:
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/SameKeyReader.java
##
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import java.io.InputStream;
+import java.security.MessageDigest;
+import java.util.concurrent.Callable;
+
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.OzoneClientFactory;
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.io.IOUtils;
+import picocli.CommandLine.Command;
+import picocli.CommandLine.Option;
+
+/**
+ * Data generator tool test om performance.
+ */
+@Command(name = "ocokr",
+aliases = "ozone-client-one-key-reader",
+description = "Read the same key from multiple threads.",
+versionProvider = HddsVersionProvider.class,
+mixinStandardHelpOptions = true,
+showDefaultValues = true)
+public class SameKeyReader extends BaseFreonGenerator
+implements Callable {
+
+ @Option(names = {"-v", "--volume"},
+ description = "Name of the bucket which contains the test data. Will be"
+ + " created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data. Will be"
+ + " created if missing.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @Option(names = {"-k", "--key"},
+ description = "Name of the key read from multiple threads",
+ defaultValue = "bucket1")
Review comment:
Oh, Shame on me. I didn't notice that it's a **key** and not a bucket. Let
me make it required without default value.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] elek closed pull request #1439: HDDS-2125. maven-javadoc-plugin.version is missing in pom.ozone.xml.
elek closed pull request #1439: HDDS-2125. maven-javadoc-plugin.version is missing in pom.ozone.xml. URL: https://github.com/apache/hadoop/pull/1439 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1446: YARN-9834. Allow using a pool of local users to run Yarn Secure Conta…
hadoop-yetus commented on issue #1446: YARN-9834. Allow using a pool of local users to run Yarn Secure Conta… URL: https://github.com/apache/hadoop/pull/1446#issuecomment-531447976 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 2157 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 45 | Maven dependency ordering for branch | | +1 | mvninstall | 1113 | trunk passed | | +1 | compile | 557 | trunk passed | | +1 | checkstyle | 90 | trunk passed | | +1 | mvnsite | 110 | trunk passed | | +1 | shadedclient | 892 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 88 | trunk passed | | 0 | spotbugs | 93 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 208 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 19 | Maven dependency ordering for patch | | -1 | mvninstall | 24 | hadoop-yarn-server-nodemanager in the patch failed. | | -1 | compile | 79 | hadoop-yarn in the patch failed. | | -1 | javac | 79 | hadoop-yarn in the patch failed. | | -0 | checkstyle | 73 | hadoop-yarn-project/hadoop-yarn: The patch generated 19 new + 382 unchanged - 1 fixed = 401 total (was 383) | | -1 | mvnsite | 25 | hadoop-yarn-server-nodemanager in the patch failed. | | +1 | whitespace | 0 | The patch has no whitespace issues. | | -1 | shadedclient | 265 | patch has errors when building and testing our client artifacts. | | -1 | javadoc | 28 | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 9 new + 0 unchanged - 0 fixed = 9 total (was 0) | | -1 | findbugs | 27 | hadoop-yarn-server-nodemanager in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 52 | hadoop-yarn-api in the patch failed. | | -1 | unit | 26 | hadoop-yarn-server-nodemanager in the patch failed. | | -1 | asflicense | 31 | The patch generated 1 ASF License warnings. | | | | 6146 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1446/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1446 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux eaaeab1ea91e 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 6a9f7ca | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1446/1/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1446/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1446/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1446/1/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt | | mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1446/1/artifact/out/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1446/1/artifact/out/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1446/1/artifact/out/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1446/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1446/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1446/1/testReport/ | | asflicense | https://builds.apache.org/job/hadoop-multibranch/job/PR-1446/1/artifact/out/patch-asflicense-problems.txt | | Max. process+thread count | 415 (vs. ulimit of
[GitHub] [hadoop] elek opened a new pull request #1448: HDDS-2110. Arbitrary file can be downloaded with the help of ProfilerServlet
elek opened a new pull request #1448: HDDS-2110. Arbitrary file can be
downloaded with the help of ProfilerServlet
URL: https://github.com/apache/hadoop/pull/1448
The LOC 324 in the file
[ProfileServlet.java|https://github.com/apache/hadoop/blob/217bdbd940a96986df3b96899b43caae2b5a9ed2/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/ProfileServlet.java]
is prone to an arbitrary file download:-
{code:java}
protected void doGetDownload(String fileName, final HttpServletRequest req,
final HttpServletResponse resp) throws IOException {
File requestedFile =
ProfileServlet.OUTPUT_DIR.resolve(fileName).toAbsolutePath().toFile();{code}
As the String fileName is directly considered as the requested file.
Which is called at LOC 180 with HTTP request directly passed:-
{code:java}
if (req.getParameter("file") != null) {
doGetDownload(req.getParameter("file"), req, resp);
return;
}
{code}
See: https://issues.apache.org/jira/browse/HDDS-2110
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1447: HDDS-2111. XSS fragments can be injected to the S3g landing page
hadoop-yetus commented on issue #1447: HDDS-2111. XSS fragments can be injected to the S3g landing page URL: https://github.com/apache/hadoop/pull/1447#issuecomment-531447108 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 42 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | -1 | mvninstall | 32 | hadoop-ozone in trunk failed. | | -1 | compile | 21 | hadoop-ozone in trunk failed. | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 880 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 21 | hadoop-hdds in trunk failed. | | -1 | javadoc | 19 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | -1 | mvninstall | 34 | hadoop-ozone in the patch failed. | | -1 | jshint | 77 | The patch generated 1392 new + 2737 unchanged - 0 fixed = 4129 total (was 2737) | | -1 | compile | 25 | hadoop-ozone in the patch failed. | | -1 | javac | 25 | hadoop-ozone in the patch failed. | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 654 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 18 | hadoop-hdds in the patch failed. | | -1 | javadoc | 18 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 134 | hadoop-hdds in the patch failed. | | -1 | unit | 28 | hadoop-ozone in the patch failed. | | +1 | asflicense | 32 | The patch does not generate ASF License warnings. | | | | 2408 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.ozoneimpl.TestOzoneContainer | | | hadoop.ozone.container.keyvalue.TestKeyValueContainer | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1447/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1447 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient jshint | | uname | Linux f1154b87d514 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 6a9f7ca | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1447/1/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1447/1/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1447/1/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1447/1/artifact/out/branch-javadoc-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1447/1/artifact/out/patch-mvninstall-hadoop-ozone.txt | | jshint | https://builds.apache.org/job/hadoop-multibranch/job/PR-1447/1/artifact/out/diff-patch-jshint.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1447/1/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1447/1/artifact/out/patch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1447/1/artifact/out/patch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1447/1/artifact/out/patch-javadoc-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1447/1/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1447/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1447/1/testReport/ | | Max. process+thread count | 399 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/s3gateway U: hadoop-ozone/s3gateway | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1447/1/console | | versions | git=2.7.4 maven=3.3.9 jshint=2.10.2 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated
[GitHub] [hadoop] hadoop-yetus commented on issue #1277: HDDS-1054. List Multipart uploads in a bucket
hadoop-yetus commented on issue #1277: HDDS-1054. List Multipart uploads in a bucket URL: https://github.com/apache/hadoop/pull/1277#issuecomment-531446769 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 1208 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 77 | Maven dependency ordering for branch | | -1 | mvninstall | 32 | hadoop-ozone in trunk failed. | | -1 | compile | 19 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 100 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 937 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 13 | hadoop-hdds in trunk failed. | | -1 | javadoc | 15 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 159 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 23 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 23 | Maven dependency ordering for patch | | -1 | mvninstall | 31 | hadoop-ozone in the patch failed. | | -1 | compile | 22 | hadoop-ozone in the patch failed. | | -1 | cc | 22 | hadoop-ozone in the patch failed. | | -1 | javac | 22 | hadoop-ozone in the patch failed. | | -0 | checkstyle | 24 | hadoop-hdds: The patch generated 5 new + 9 unchanged - 1 fixed = 14 total (was 10) | | -0 | checkstyle | 71 | hadoop-ozone: The patch generated 373 new + 2410 unchanged - 15 fixed = 2783 total (was 2425) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 1 | The patch has no whitespace issues. | | +1 | shadedclient | 726 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 14 | hadoop-hdds in the patch failed. | | -1 | javadoc | 15 | hadoop-ozone in the patch failed. | | -1 | findbugs | 23 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 163 | hadoop-hdds in the patch failed. | | -1 | unit | 25 | hadoop-ozone in the patch failed. | | +1 | asflicense | 29 | The patch does not generate ASF License warnings. | | | | 4264 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.ozoneimpl.TestOzoneContainer | | | hadoop.ozone.container.keyvalue.TestKeyValueContainer | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1277 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux 86494a8d4639 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 6a9f7ca | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/patch-compile-hadoop-ozone.txt | | cc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/patch-compile-hadoop-ozone.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/diff-checkstyle-hadoop-hdds.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/diff-checkstyle-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/patch-javadoc-hadoop-hdds.txt | | javadoc |
[GitHub] [hadoop] elek opened a new pull request #1447: HDDS-2111. XSS fragments can be injected to the S3g landing page
elek opened a new pull request #1447: HDDS-2111. XSS fragments can be injected
to the S3g landing page
URL: https://github.com/apache/hadoop/pull/1447
VULNERABILITY DETAILS
There is a way to bypass anti-XSS filter for DOM XSS exploiting a
"window.location.href".
Considering a typical URL:
scheme://domain:port/path?query_string#fragment_id
Browsers encode correctly both "path" and "query_string", but not the
"fragment_id".
So if used "fragment_id" the vector is also not logged on Web Server.
VERSION
Chrome Version: 10.0.648.134 (Official Build 77917) beta
REPRODUCTION CASE
This is an index.html page:
{code:java}
aws s3api --endpoint
document.write(window.location.href.replace("static/", ""))
create-bucket --bucket=wordcount
{code}
The attack vector is:
index.html?#alert('XSS');
* PoC:
For your convenience, a minimalist PoC is located on:
http://security.onofri.org/xss_location.html?#alert('XSS');
* References
- DOM Based Cross-Site Scripting or XSS of the Third Kind -
http://www.webappsec.org/projects/articles/071105.shtml
reference:-
https://bugs.chromium.org/p/chromium/issues/detail?id=76796
See: https://issues.apache.org/jira/browse/HDDS-2111
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] elek commented on a change in pull request #1277: HDDS-1054. List Multipart uploads in a bucket
elek commented on a change in pull request #1277: HDDS-1054. List Multipart
uploads in a bucket
URL: https://github.com/apache/hadoop/pull/1277#discussion_r324409675
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
##
@@ -1270,6 +1271,58 @@ public void abortMultipartUpload(OmKeyArgs omKeyArgs)
throws IOException {
}
+ @Override
+ public OmMultipartUploadList listMultipartUploads(String volumeName,
+ String bucketName, String prefix) throws OMException {
+Preconditions.checkNotNull(volumeName);
+Preconditions.checkNotNull(bucketName);
+
+metadataManager.getLock().acquireLock(BUCKET_LOCK, volumeName, bucketName);
+try {
+
+ List multipartUploadKeys =
+ metadataManager
+ .getMultipartUploadKeys(volumeName, bucketName, prefix);
+
+ List collect = multipartUploadKeys.stream()
+ .map(OmMultipartUpload::from)
+ .map(upload -> {
+String dbKey = metadataManager
+.getOzoneKey(upload.getVolumeName(),
+upload.getBucketName(),
+upload.getKeyName());
+try {
+ Table openKeyTable =
+ metadataManager.getOpenKeyTable();
+
+ OmKeyInfo omKeyInfo =
+ openKeyTable.get(upload.getDbKey());
Review comment:
You are 100% right. But it seems to be a bigger change. Let's do it in
https://issues.apache.org/jira/browse/HDDS-2131
(On the other hand I added audit + metrics support because they were just a
few lines)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] elek commented on issue #1277: HDDS-1054. List Multipart uploads in a bucket
elek commented on issue #1277: HDDS-1054. List Multipart uploads in a bucket URL: https://github.com/apache/hadoop/pull/1277#issuecomment-531442173 > Have few comments in place. > Can we open Jira's for all TODO's in this Jira for tracking purpose? > > 1. Audit support for the new method. > > 2. Pagination support and other query parameters support. > > 3. If replication type will not be handled in this Jira, can you open Jira for this one too. Sure. Audit + the replication type support is added to this patch. I created HDDS-2130 for the pagination support. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] elek commented on a change in pull request #1277: HDDS-1054. List Multipart uploads in a bucket
elek commented on a change in pull request #1277: HDDS-1054. List Multipart
uploads in a bucket
URL: https://github.com/apache/hadoop/pull/1277#discussion_r324409356
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java
##
@@ -826,6 +828,28 @@ private VolumeList getVolumesByUser(String userNameKey)
return count;
}
+ @Override
+ public List getMultipartUploadKeys(
+ String volumeName, String bucketName, String prefix) throws IOException {
+List response = new ArrayList<>();
+TableIterator>
+iterator = getMultipartInfoTable().iterator();
+
+String prefixKey =
+OmMultipartUpload.getDbKey(volumeName, bucketName, prefix);
+iterator.seek(prefixKey);
+
+while (iterator.hasNext()) {
Review comment:
Can you please help me to understand how should it be done? What about if an
MPU is finished and deleted? How is it cached? I think I can't return with
(cached values + db values) because the deletions
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] shanyu opened a new pull request #1446: YARN-9834. Allow using a pool of local users to run Yarn Secure Conta…
shanyu opened a new pull request #1446: YARN-9834. Allow using a pool of local users to run Yarn Secure Conta… URL: https://github.com/apache/hadoop/pull/1446 …iner in secure mode Signed-off-by: Shanyu Zhao https://issues.apache.org/jira/browse/YARN-9834 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] elek commented on a change in pull request #1277: HDDS-1054. List Multipart uploads in a bucket
elek commented on a change in pull request #1277: HDDS-1054. List Multipart uploads in a bucket URL: https://github.com/apache/hadoop/pull/1277#discussion_r324409058 ## File path: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMMetadataManager.java ## @@ -327,4 +327,11 @@ String getMultipartKey(String volume, String bucket, String key, String */ long countEstimatedRowsInTable(Table table) throws IOException; + + /** + * Return the existing upload keys which includes volumeName, bucketName, + * keyName and uploadId. Review comment: Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15578) GridmixTestUtils uses the wrong staging directory in windows
[
https://issues.apache.org/jira/browse/HADOOP-15578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929621#comment-16929621
]
Hadoop QA commented on HADOOP-15578:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 37m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 10s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 42s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m
42s{color} | {color:green} hadoop-gridmix in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}100m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:39e82acc485 |
| JIRA Issue | HADOOP-15578 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12930048/HADOOP-15578.000.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 7bb166d9d0d4 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6a9f7ca |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16528/testReport/ |
| Max. process+thread count | 981 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-gridmix U: hadoop-tools/hadoop-gridmix |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16528/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> GridmixTestUtils uses the wrong staging directory in windows
>
>
> Key: HADOOP-15578
>
[jira] [Commented] (HADOOP-12928) Update netty to 3.10.5.Final to sync with zookeeper
[ https://issues.apache.org/jira/browse/HADOOP-12928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929617#comment-16929617 ] Jonathan Hung commented on HADOOP-12928: Hi [~ozawa] / [~eddyxu], I see this never got committed to branch-2, any reason for that? There was a ticket filed for upgrading netty in branch-2 here: HADOOP-16544. Thanks! > Update netty to 3.10.5.Final to sync with zookeeper > --- > > Key: HADOOP-12928 > URL: https://issues.apache.org/jira/browse/HADOOP-12928 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.7.2 >Reporter: Hendy Irawan >Assignee: Lei (Eddy) Xu >Priority: Major > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-12928-branch-2.00.patch, > HADOOP-12928-branch-2.01.patch, HADOOP-12928-branch-2.02.patch, > HADOOP-12928.01.patch, HADOOP-12928.02.patch, HADOOP-12928.03.patch, > HDFS-12928.00.patch > > > Update netty to 3.7.1.Final because hadoop-client 2.7.2 depends on zookeeper > 3.4.6 which depends on netty 3.7.x. Related to HADOOP-12927 > Pull request: https://github.com/apache/hadoop/pull/85 -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1445: HDDS-2128. Make ozone sh command work with OM HA service ids
hadoop-yetus commented on issue #1445: HDDS-2128. Make ozone sh command work with OM HA service ids URL: https://github.com/apache/hadoop/pull/1445#issuecomment-531422743 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 40 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | -1 | mvninstall | 31 | hadoop-ozone in trunk failed. | | -1 | compile | 22 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 51 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 814 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 18 | hadoop-hdds in trunk failed. | | -1 | javadoc | 18 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 146 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 26 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | -1 | mvninstall | 33 | hadoop-ozone in the patch failed. | | -1 | compile | 25 | hadoop-ozone in the patch failed. | | -1 | javac | 25 | hadoop-ozone in the patch failed. | | +1 | checkstyle | 56 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 650 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 17 | hadoop-hdds in the patch failed. | | -1 | javadoc | 18 | hadoop-ozone in the patch failed. | | -1 | findbugs | 27 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 127 | hadoop-hdds in the patch failed. | | -1 | unit | 28 | hadoop-ozone in the patch failed. | | +1 | asflicense | 34 | The patch does not generate ASF License warnings. | | | | 2735 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.keyvalue.TestKeyValueContainer | | | hadoop.ozone.container.ozoneimpl.TestOzoneContainer | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1445/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1445 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 60fc3e546740 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 6a9f7ca | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1445/1/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1445/1/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1445/1/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1445/1/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1445/1/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1445/1/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1445/1/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1445/1/artifact/out/patch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1445/1/artifact/out/patch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1445/1/artifact/out/patch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1445/1/artifact/out/patch-findbugs-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1445/1/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1445/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1445/1/testReport/ | | Max. process+thread count | 411 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/ozone-manager U: hadoop-ozone/ozone-manager | |
[GitHub] [hadoop] xiaoyuyao commented on a change in pull request #1440: HDDS-2114: Rename does not preserve non-explicitly created interim directories
xiaoyuyao commented on a change in pull request #1440: HDDS-2114: Rename does
not preserve non-explicitly created interim directories
URL: https://github.com/apache/hadoop/pull/1440#discussion_r324392898
##
File path:
hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java
##
@@ -374,7 +374,11 @@ public boolean rename(Path src, Path dst) throws
IOException {
}
}
RenameIterator iterator = new RenameIterator(src, dst);
-return iterator.iterate();
+boolean result = iterator.iterate();
+if (result) {
+ createFakeParentDirectory(src);
Review comment:
should we createFakeParentDirectory for dst as it is the rename result?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao commented on a change in pull request #1440: HDDS-2114: Rename does not preserve non-explicitly created interim directories
xiaoyuyao commented on a change in pull request #1440: HDDS-2114: Rename does
not preserve non-explicitly created interim directories
URL: https://github.com/apache/hadoop/pull/1440#discussion_r324392898
##
File path:
hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java
##
@@ -374,7 +374,11 @@ public boolean rename(Path src, Path dst) throws
IOException {
}
}
RenameIterator iterator = new RenameIterator(src, dst);
-return iterator.iterate();
+boolean result = iterator.iterate();
+if (result) {
+ createFakeParentDirectory(src);
Review comment:
should we createFakeParentDirectory for dst as it is the rename result?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] smengcl opened a new pull request #1445: HDDS-2128. Make ozone sh command work with OM HA service ids
smengcl opened a new pull request #1445: HDDS-2128. Make ozone sh command work with OM HA service ids URL: https://github.com/apache/hadoop/pull/1445 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1444: HDDS-2078. Get/Renew DelegationToken NPE after HDDS-1909. Contributed…
hadoop-yetus commented on issue #1444: HDDS-2078. Get/Renew DelegationToken NPE after HDDS-1909. Contributed… URL: https://github.com/apache/hadoop/pull/1444#issuecomment-531410808 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 1117 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | -1 | mvninstall | 30 | hadoop-ozone in trunk failed. | | -1 | compile | 23 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 48 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 808 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 17 | hadoop-hdds in trunk failed. | | -1 | javadoc | 18 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 148 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 25 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | -1 | mvninstall | 32 | hadoop-ozone in the patch failed. | | -1 | compile | 25 | hadoop-ozone in the patch failed. | | -1 | javac | 25 | hadoop-ozone in the patch failed. | | +1 | checkstyle | 55 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 665 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 18 | hadoop-hdds in the patch failed. | | -1 | javadoc | 18 | hadoop-ozone in the patch failed. | | -1 | findbugs | 27 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 130 | hadoop-hdds in the patch failed. | | -1 | unit | 29 | hadoop-ozone in the patch failed. | | +1 | asflicense | 32 | The patch does not generate ASF License warnings. | | | | 3815 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.keyvalue.TestKeyValueContainer | | | hadoop.ozone.container.ozoneimpl.TestOzoneContainer | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1444/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1444 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 67e4ed679fe3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 6a9f7ca | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1444/1/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1444/1/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1444/1/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1444/1/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1444/1/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1444/1/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1444/1/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1444/1/artifact/out/patch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1444/1/artifact/out/patch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1444/1/artifact/out/patch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1444/1/artifact/out/patch-findbugs-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1444/1/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1444/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1444/1/testReport/ | | Max. process+thread count | 411 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/ozone-manager U: hadoop-ozone/ozone-manager |
[jira] [Commented] (HADOOP-15578) GridmixTestUtils uses the wrong staging directory in windows
[
https://issues.apache.org/jira/browse/HADOOP-15578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929564#comment-16929564
]
Ayush Saxena commented on HADOOP-15578:
---
Thanx [~elgoiri] for the report, I guess this problem still persists.
bq. Another option is to sanitize the Windows path to make it usable for HDFS.
IMO, if we can do this, should be better, we can use the configured value,
rather than ignoring it.
> GridmixTestUtils uses the wrong staging directory in windows
>
>
> Key: HADOOP-15578
> URL: https://issues.apache.org/jira/browse/HADOOP-15578
> Project: Hadoop Common
> Issue Type: Test
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: HADOOP-15578.000.patch
>
>
> {{GridmixTestUtils#createHomeAndStagingDirectory}} gets the staging area from
> the configuration key {{mapreduce.jobtracker.staging.root.dir}}. This
> variable depends on {{hadoop.tmp.dir}} which in Windows is set to a local
> Windows folder. When the test tries to create the path in HDFS it gets an
> error because the path is not compliant.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao opened a new pull request #1444: HDDS-2078. Get/Renew DelegationToken NPE after HDDS-1909. Contributed…
xiaoyuyao opened a new pull request #1444: HDDS-2078. Get/Renew DelegationToken NPE after HDDS-1909. Contributed… URL: https://github.com/apache/hadoop/pull/1444 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1443: HDDS-2124. Random next links
hadoop-yetus commented on issue #1443: HDDS-2124. Random next links URL: https://github.com/apache/hadoop/pull/1443#issuecomment-531369389 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 1159 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | ||| _ trunk Compile Tests _ | | -1 | mvninstall | 34 | hadoop-ozone in trunk failed. | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 912 | branch has no errors when building and testing our client artifacts. | ||| _ Patch Compile Tests _ | | -1 | mvninstall | 34 | hadoop-ozone in the patch failed. | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 680 | patch has no errors when building and testing our client artifacts. | ||| _ Other Tests _ | | +1 | asflicense | 34 | The patch does not generate ASF License warnings. | | | | 3033 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1443/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1443 | | Optional Tests | dupname asflicense mvnsite | | uname | Linux 659f01becfde 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 6a9f7ca | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1443/1/artifact/out/branch-mvninstall-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1443/1/artifact/out/patch-mvninstall-hadoop-ozone.txt | | Max. process+thread count | 446 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/docs U: hadoop-hdds/docs | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1443/1/console | | versions | git=2.7.4 maven=3.3.9 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] smengcl commented on issue #1360: HDDS-2007. Make ozone fs shell command work with OM HA service ids
smengcl commented on issue #1360: HDDS-2007. Make ozone fs shell command work with OM HA service ids URL: https://github.com/apache/hadoop/pull/1360#issuecomment-531358343 Thanks for committing @bharatviswa504 ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1442: HADOOP-16570. S3A committers leak threads on job/task commit.
hadoop-yetus commented on issue #1442: HADOOP-16570. S3A committers leak threads on job/task commit. URL: https://github.com/apache/hadoop/pull/1442#issuecomment-531357838 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 1142 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1198 | trunk passed | | +1 | compile | 34 | trunk passed | | +1 | checkstyle | 27 | trunk passed | | +1 | mvnsite | 36 | trunk passed | | +1 | shadedclient | 777 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 26 | trunk passed | | 0 | spotbugs | 60 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 56 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 33 | the patch passed | | +1 | compile | 27 | the patch passed | | +1 | javac | 27 | the patch passed | | +1 | checkstyle | 18 | the patch passed | | +1 | mvnsite | 30 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 816 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 23 | the patch passed | | +1 | findbugs | 61 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 74 | hadoop-aws in the patch passed. | | +1 | asflicense | 28 | The patch does not generate ASF License warnings. | | | | 4502 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1442/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1442 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux abef849a9c36 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 1843c46 | | Default Java | 1.8.0_222 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1442/1/testReport/ | | Max. process+thread count | 353 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1442/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1441: HADOOP-16568. S3A FullCredentialsTokenBinding fails if local credentials are unset
hadoop-yetus commented on issue #1441: HADOOP-16568. S3A FullCredentialsTokenBinding fails if local credentials are unset URL: https://github.com/apache/hadoop/pull/1441#issuecomment-531357805 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 1178 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1196 | trunk passed | | +1 | compile | 31 | trunk passed | | +1 | checkstyle | 23 | trunk passed | | +1 | mvnsite | 36 | trunk passed | | +1 | shadedclient | 794 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 25 | trunk passed | | 0 | spotbugs | 69 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 67 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 34 | the patch passed | | +1 | compile | 33 | the patch passed | | +1 | javac | 33 | the patch passed | | +1 | checkstyle | 21 | the patch passed | | +1 | mvnsite | 40 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 858 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 26 | the patch passed | | +1 | findbugs | 71 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 84 | hadoop-aws in the patch passed. | | +1 | asflicense | 28 | The patch does not generate ASF License warnings. | | | | 4644 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1441/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1441 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 83aa939dc532 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 1843c46 | | Default Java | 1.8.0_222 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1441/1/testReport/ | | Max. process+thread count | 340 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1441/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on issue #1443: HDDS-2124. Random next links
adoroszlai commented on issue #1443: HDDS-2124. Random next links URL: https://github.com/apache/hadoop/pull/1443#issuecomment-531354016 /label ozone This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adoroszlai opened a new pull request #1443: HDDS-2124. Random next links
adoroszlai opened a new pull request #1443: HDDS-2124. Random next links URL: https://github.com/apache/hadoop/pull/1443 ## What changes were proposed in this pull request? Assign weights to pages to ensure correct order. https://issues.apache.org/jira/browse/HDDS-2124 ## How was this patch tested? Generated docs, verified order. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 edited a comment on issue #1377: HDDS-2057. Incorrect Default OM Port in Ozone FS URI Error Message. Contributed by Supratim Deka
bharatviswa504 edited a comment on issue #1377: HDDS-2057. Incorrect Default OM Port in Ozone FS URI Error Message. Contributed by Supratim Deka URL: https://github.com/apache/hadoop/pull/1377#issuecomment-531347971 I think, we need to revert this. Due to some classpath loading magic stuff, we cannot use OzoneConfiguration class in BasicOzoneFileSystem.java (That is the reason acceptance test are failing with hadoop2 tests). We have seen a similar error during HDDS-2007, it was resolved by moving logic to BasicOzoneClientAdapterImpl. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on issue #1377: HDDS-2057. Incorrect Default OM Port in Ozone FS URI Error Message. Contributed by Supratim Deka
bharatviswa504 commented on issue #1377: HDDS-2057. Incorrect Default OM Port in Ozone FS URI Error Message. Contributed by Supratim Deka URL: https://github.com/apache/hadoop/pull/1377#issuecomment-531347971 I think, we need to revert this. Due to some classpath loading magic stuff, we cannot use OzoneConfiguration class in BasicOzoneFileSystem.java. We have seen a similar error during HDDS-2007, it was resolved by moving logic to BasicOzoneClientAdapterImpl. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] vivekratnavel commented on issue #1424: HDDS-2107. Datanodes should retry forever to connect to SCM in an…
vivekratnavel commented on issue #1424: HDDS-2107. Datanodes should retry forever to connect to SCM in an… URL: https://github.com/apache/hadoop/pull/1424#issuecomment-531343596 @hanishakoneru Sure. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16555) Update commons-compress to 1.19
[ https://issues.apache.org/jira/browse/HADOOP-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-16555. -- Resolution: Fixed There was a trivial conflict cherrypicking the commits into lower branches. Uploaded [^HADOOP-16555.branch-3.2.patch] for future reference. We may have to cherry pick this into branch-2. > Update commons-compress to 1.19 > --- > > Key: HADOOP-16555 > URL: https://issues.apache.org/jira/browse/HADOOP-16555 > Project: Hadoop Common > Issue Type: Task >Reporter: Wei-Chiu Chuang >Assignee: YiSheng Lien >Priority: Major > Labels: release-blocker > Fix For: 3.3.0, 3.2.1, 3.1.3 > > Attachments: HADOOP-16555.branch-3.2.patch > > > We depend on commons-compress 1.18. The 1.19 release just went out. I think > we should update it. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16555) Update commons-compress to 1.19
[ https://issues.apache.org/jira/browse/HADOOP-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-16555: - Fix Version/s: 3.1.3 3.2.1 > Update commons-compress to 1.19 > --- > > Key: HADOOP-16555 > URL: https://issues.apache.org/jira/browse/HADOOP-16555 > Project: Hadoop Common > Issue Type: Task >Reporter: Wei-Chiu Chuang >Assignee: YiSheng Lien >Priority: Major > Labels: release-blocker > Fix For: 3.3.0, 3.2.1, 3.1.3 > > Attachments: HADOOP-16555.branch-3.2.patch > > > We depend on commons-compress 1.18. The 1.19 release just went out. I think > we should update it. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on issue #1360: HDDS-2007. Make ozone fs shell command work with OM HA service ids
bharatviswa504 commented on issue #1360: HDDS-2007. Make ozone fs shell command work with OM HA service ids URL: https://github.com/apache/hadoop/pull/1360#issuecomment-531341578 Thank You @smengcl for the contribution. I have committed this to the trunk. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 merged pull request #1360: HDDS-2007. Make ozone fs shell command work with OM HA service ids
bharatviswa504 merged pull request #1360: HDDS-2007. Make ozone fs shell command work with OM HA service ids URL: https://github.com/apache/hadoop/pull/1360 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16555) Update commons-compress to 1.19
[ https://issues.apache.org/jira/browse/HADOOP-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-16555: - Attachment: HADOOP-16555.branch-3.2.patch > Update commons-compress to 1.19 > --- > > Key: HADOOP-16555 > URL: https://issues.apache.org/jira/browse/HADOOP-16555 > Project: Hadoop Common > Issue Type: Task >Reporter: Wei-Chiu Chuang >Assignee: YiSheng Lien >Priority: Major > Labels: release-blocker > Fix For: 3.3.0 > > Attachments: HADOOP-16555.branch-3.2.patch > > > We depend on commons-compress 1.18. The 1.19 release just went out. I think > we should update it. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16555) Update commons-compress to 1.19
[ https://issues.apache.org/jira/browse/HADOOP-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-16555: - Fix Version/s: 3.3.0 > Update commons-compress to 1.19 > --- > > Key: HADOOP-16555 > URL: https://issues.apache.org/jira/browse/HADOOP-16555 > Project: Hadoop Common > Issue Type: Task >Reporter: Wei-Chiu Chuang >Assignee: YiSheng Lien >Priority: Major > Labels: release-blocker > Fix For: 3.3.0 > > > We depend on commons-compress 1.18. The 1.19 release just went out. I think > we should update it. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16555) Update commons-compress to 1.19
[ https://issues.apache.org/jira/browse/HADOOP-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929416#comment-16929416 ] Hudson commented on HADOOP-16555: - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17295 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17295/]) HADOOP-16555. Update commons-compress to 1.19. (#1425) Contributed by (weichiu: rev 1843c4688a3040c48f6834d166494cbf68b5e4fd) * (edit) LICENSE-binary * (edit) hadoop-project/pom.xml > Update commons-compress to 1.19 > --- > > Key: HADOOP-16555 > URL: https://issues.apache.org/jira/browse/HADOOP-16555 > Project: Hadoop Common > Issue Type: Task >Reporter: Wei-Chiu Chuang >Assignee: YiSheng Lien >Priority: Major > Labels: release-blocker > > We depend on commons-compress 1.18. The 1.19 release just went out. I think > we should update it. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16555) Update commons-compress to 1.19
[ https://issues.apache.org/jira/browse/HADOOP-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reassigned HADOOP-16555: Assignee: YiSheng Lien > Update commons-compress to 1.19 > --- > > Key: HADOOP-16555 > URL: https://issues.apache.org/jira/browse/HADOOP-16555 > Project: Hadoop Common > Issue Type: Task >Reporter: Wei-Chiu Chuang >Assignee: YiSheng Lien >Priority: Major > Labels: release-blocker > > We depend on commons-compress 1.18. The 1.19 release just went out. I think > we should update it. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang merged pull request #1425: HADOOP-16555. Update commons-compress to 1.19.
jojochuang merged pull request #1425: HADOOP-16555. Update commons-compress to 1.19. URL: https://github.com/apache/hadoop/pull/1425 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on issue #1425: HADOOP-16555. Update commons-compress to 1.19.
jojochuang commented on issue #1425: HADOOP-16555. Update commons-compress to 1.19. URL: https://github.com/apache/hadoop/pull/1425#issuecomment-531337509 For the record, here are the two failed tests: > org.apache.hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy.testNNSendsErasureCodingTasks > org.apache.hadoop.hdfs.qjournal.client.TestQJMWithFaults.testRecoverAfterDoubleFailures This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on issue #1425: HADOOP-16555. Update commons-compress to 1.19.
jojochuang commented on issue #1425: HADOOP-16555. Update commons-compress to 1.19. URL: https://github.com/apache/hadoop/pull/1425#issuecomment-531337110 +1 The precommit had two test failures, both looks unrelated. However, it didn't report back to GitHub: > Could not update commit status, please check if your scan credentials belong to a member of the organization or a collaborator of the repository and repo:status scope is selected > > GitHub has been notified of this commit’s build result This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-16570) S3A committers leak threads on job/task commit
[ https://issues.apache.org/jira/browse/HADOOP-16570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-16570 started by Steve Loughran. --- > S3A committers leak threads on job/task commit > -- > > Key: HADOOP-16570 > URL: https://issues.apache.org/jira/browse/HADOOP-16570 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0, 3.1.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > The fixed size ThreadPool created in AbstractS3ACommitter doesn't get cleaned > up at EOL; as a result you leak the no. of threads set in > "fs.s3a.committer.threads" > Not visible in MR/distcp jobs, but ultimately causes OOM on Spark -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1442: HADOOP-16570. S3A committers leak threads on job/task commit.
steveloughran commented on issue #1442: HADOOP-16570. S3A committers leak threads on job/task commit. URL: https://github.com/apache/hadoop/pull/1442#issuecomment-531333276 tested s3 ireland, ddb, auth This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran opened a new pull request #1442: HADOOP-16570. S3A committers leak threads on job/task commit.
steveloughran opened a new pull request #1442: HADOOP-16570. S3A committers leak threads on job/task commit. URL: https://github.com/apache/hadoop/pull/1442 This patch explicitly shuts down the thread pool in job cleanup and after task commit, abort, job abort and job commit. The alternative strategy would to be to always destroy the threads in the same method they were used, but as two operations are normally parallelized back-to-back: listing pending .files and then committing or aborting them, retaining the pool is useful. And there isn't any close() method or similar in the OutputCommitter interface to place it. To test this, a probe for the committer having a thread pool was added, and the AbstractITCommitProtocol test extended to verify that there was no thread pool after the various commit and abort lifecycle operations. To verify that the tests themselves were valid, the destroyThreadPool() initially *did not* actually destroy the pool; the fact that the modified tests then all failed providesd evidence that all paths followed in those tests successfully cleaned up. Once the method did close the thread pool, all these failing tests passed. Change-Id: Ib2765d70aae2658535e07da268899d72824094f4 Note: I also switched to the HadoopExecutors thread pool factory; I considered moving to one of the caching thread pools but decided that I'd make this change simpler for ease of backport. For a trunk-only fix I'd consider asking the target S3A FS for its store context and creating a thread pool of it, which would just be a restricted fraction of the store's own pool. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1441: HADOOP-16568. S3A FullCredentialsTokenBinding fails if local credentials are unset
steveloughran commented on issue #1441: HADOOP-16568. S3A FullCredentialsTokenBinding fails if local credentials are unset URL: https://github.com/apache/hadoop/pull/1441#issuecomment-531332720 tested: ITests and manually. Obviously those itest tests aren't enough, but I couldn't think of a simple way to verify this This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran opened a new pull request #1441: HADOOP-16568. S3A FullCredentialsTokenBinding fails if local credentials are unset
steveloughran opened a new pull request #1441: HADOOP-16568. S3A FullCredentialsTokenBinding fails if local credentials are unset URL: https://github.com/apache/hadoop/pull/1441 Move the local loading to deployUnbonded() (where they are required) and add a safety check when a new DT is requested Change-Id: I516368c2c4a558a2a86e8cf107f77f1e40338261 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bshashikant commented on issue #1163: HDDS-1786 : Datanodes takeSnapshot should delete previously created s…
bshashikant commented on issue #1163: HDDS-1786 : Datanodes takeSnapshot should delete previously created s… URL: https://github.com/apache/hadoop/pull/1163#issuecomment-531331231 Thanks @avijayanhwx for the contribution. I have committed this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bshashikant merged pull request #1163: HDDS-1786 : Datanodes takeSnapshot should delete previously created s…
bshashikant merged pull request #1163: HDDS-1786 : Datanodes takeSnapshot should delete previously created s… URL: https://github.com/apache/hadoop/pull/1163 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1369: HDDS-2020. Remove mTLS from Ozone GRPC. Contributed by Xiaoyu Yao.
hadoop-yetus commented on issue #1369: HDDS-2020. Remove mTLS from Ozone GRPC. Contributed by Xiaoyu Yao. URL: https://github.com/apache/hadoop/pull/1369#issuecomment-531320049 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 1174 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 2 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 11 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 68 | Maven dependency ordering for branch | | -1 | mvninstall | 41 | hadoop-ozone in trunk failed. | | -1 | compile | 22 | hadoop-ozone in trunk failed. | | -0 | checkstyle | 133 | The patch fails to run checkstyle in hadoop-ozone | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 950 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 17 | hadoop-hdds in trunk failed. | | -1 | javadoc | 18 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 153 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 26 | hadoop-ozone in trunk failed. | | -0 | patch | 185 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 30 | Maven dependency ordering for patch | | -1 | mvninstall | 71 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 32 | hadoop-ozone in the patch failed. | | -1 | compile | 46 | hadoop-hdds in the patch failed. | | -1 | compile | 25 | hadoop-ozone in the patch failed. | | -1 | cc | 46 | hadoop-hdds in the patch failed. | | -1 | cc | 25 | hadoop-ozone in the patch failed. | | -1 | javac | 46 | hadoop-hdds in the patch failed. | | -1 | javac | 25 | hadoop-ozone in the patch failed. | | -0 | checkstyle | 51 | hadoop-hdds: The patch generated 69 new + 881 unchanged - 47 fixed = 950 total (was 928) | | -0 | checkstyle | 54 | The patch fails to run checkstyle in hadoop-ozone | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 2 | The patch has no ill-formed XML file. | | +1 | shadedclient | 672 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 17 | hadoop-hdds in the patch failed. | | -1 | javadoc | 18 | hadoop-ozone in the patch failed. | | -1 | findbugs | 97 | hadoop-hdds in the patch failed. | | -1 | findbugs | 26 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 120 | hadoop-hdds in the patch failed. | | -1 | unit | 28 | hadoop-ozone in the patch failed. | | +1 | asflicense | 32 | The patch does not generate ASF License warnings. | | | | 4157 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1369/7/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1369 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc | | uname | Linux e03c1c1a4ec7 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 95010a4 | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1369/7/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1369/7/artifact/out/branch-compile-hadoop-ozone.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1369/7/artifact/out//home/jenkins/jenkins-slave/workspace/hadoop-multibranch_PR-1369/out/maven-branch-checkstyle-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1369/7/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1369/7/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1369/7/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1369/7/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1369/7/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1369/7/artifact/out/patch-compile-hadoop-hdds.txt | | compile |
[GitHub] [hadoop] hanishakoneru commented on issue #1424: HDDS-2107. Datanodes should retry forever to connect to SCM in an…
hanishakoneru commented on issue #1424: HDDS-2107. Datanodes should retry forever to connect to SCM in an… URL: https://github.com/apache/hadoop/pull/1424#issuecomment-531314877 Thank you @vivekratnavel for working on this. LGTM. +1. Can you please update the description. It states that DNs fail after 10 retries which is not the case. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] arp7 commented on issue #1434: HDDS-2120. Remove hadoop classes from ozonefs-current jar
arp7 commented on issue #1434: HDDS-2120. Remove hadoop classes from ozonefs-current jar URL: https://github.com/apache/hadoop/pull/1434#issuecomment-531310413 Thanks for the quick fix @elek. Should we also exclude Ratis.. since HBase potentially has a dependency on Ratis and we should use the in-build version instead. What about third party dependencies? Downstream components could run into conflicts.. I believe that will require shading. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao commented on issue #1369: HDDS-2020. Remove mTLS from Ozone GRPC. Contributed by Xiaoyu Yao.
xiaoyuyao commented on issue #1369: HDDS-2020. Remove mTLS from Ozone GRPC. Contributed by Xiaoyu Yao. URL: https://github.com/apache/hadoop/pull/1369#issuecomment-531301387 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-16565) Fix "com.amazonaws.SdkClientException: Unable to find a region via the region provider chain."
[
https://issues.apache.org/jira/browse/HADOOP-16565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota reopened HADOOP-16565:
-
Reopened to add a message.
> Fix "com.amazonaws.SdkClientException: Unable to find a region via the region
> provider chain."
> --
>
> Key: HADOOP-16565
> URL: https://issues.apache.org/jira/browse/HADOOP-16565
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> The error found during testing in the following tests:
> {noformat}
> [ERROR] ITestS3ATemporaryCredentials.testInvalidSTSBinding:257 ? SdkClient
> Unable to f...
> [ERROR] ITestS3ATemporaryCredentials.testSTS:130 ? SdkClient Unable to find
> a region v...
> [ERROR]
> ITestS3ATemporaryCredentials.testSessionRequestExceptionTranslation:441->lambda$testSessionRequestExceptionTranslation$5:442
> ? SdkClient
> [ERROR] ITestS3ATemporaryCredentials.testSessionTokenExpiry:222 ? SdkClient
> Unable to ...
> [ERROR] ITestS3ATemporaryCredentials.testSessionTokenPropagation:193 ?
> SdkClient Unabl...
> [ERROR] ITestDelegatedMRJob.testJobSubmissionCollectsTokens:286 ? SdkClient
> Unable to ...
> [ERROR] ITestSessionDelegationInFileystem.testAddTokensFromFileSystem:235 ?
> SdkClient ...
> [ERROR]
> ITestSessionDelegationInFileystem.testCanRetrieveTokenFromCurrentUserCreds:260->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR]
> ITestSessionDelegationInFileystem.testDTCredentialProviderFromCurrentUserCreds:278->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR]
> ITestSessionDelegationInFileystem.testDelegatedFileSystem:308->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR]
> ITestSessionDelegationInFileystem.testDelegationBindingMismatch1:432->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR] ITestSessionDelegationInFileystem.testFileSystemBoundToCreator:681
> ? SdkClient
> [ERROR] ITestSessionDelegationInFileystem.testGetDTfromFileSystem:212 ?
> SdkClient Unab...
> [ERROR]
> ITestSessionDelegationInFileystem.testHDFSFetchDTCommand:606->lambda$testHDFSFetchDTCommand$3:607
> ? SdkClient
> [ERROR] ITestSessionDelegationInFileystem.testYarnCredentialPickup:576 ?
> SdkClient Una...
> [ERROR] ITestSessionDelegationTokens.testCreateAndUseDT:176 ? SdkClient
> Unable to find...
> [ERROR] ITestSessionDelegationTokens.testSaveLoadTokens:121 ? SdkClient
> Unable to find...
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1402: HADOOP-16547. make sure that s3guard prune sets up the FS
hadoop-yetus removed a comment on issue #1402: HADOOP-16547. make sure that s3guard prune sets up the FS URL: https://github.com/apache/hadoop/pull/1402#issuecomment-530779279 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 155 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1532 | trunk passed | | +1 | compile | 41 | trunk passed | | +1 | checkstyle | 38 | trunk passed | | +1 | mvnsite | 49 | trunk passed | | +1 | shadedclient | 1002 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 35 | trunk passed | | 0 | spotbugs | 76 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 70 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 41 | the patch passed | | +1 | compile | 35 | the patch passed | | +1 | javac | 35 | the patch passed | | +1 | checkstyle | 23 | the patch passed | | +1 | mvnsite | 40 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 993 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 29 | the patch passed | | +1 | findbugs | 74 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 88 | hadoop-aws in the patch passed. | | +1 | asflicense | 34 | The patch does not generate ASF License warnings. | | | | 4382 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1402/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1402 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 599076409e47 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 44850f6 | | Default Java | 1.8.0_222 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1402/3/testReport/ | | Max. process+thread count | 323 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1402/3/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16576) ITestS3GuardDDBRootOperations. test_100_FilesystemPrune failure
Steve Loughran created HADOOP-16576: --- Summary: ITestS3GuardDDBRootOperations. test_100_FilesystemPrune failure Key: HADOOP-16576 URL: https://issues.apache.org/jira/browse/HADOOP-16576 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3, test Affects Versions: 3.3.0 Reporter: Steve Loughran Failure in test_100_FilesystemPrune(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardDDBRootOperations) : No DynamoDB table name configuredjj fs.s3a.s3guard.ddb.region =eu-west-1; no region is defined. This is surfacing on a branch which doesn't have my pending prune init code, so even though an FS was passed in, it wasn't used for init. Assumption: this will go away forever when that patch is in -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16570) S3A committers leak threads on job/task commit
[ https://issues.apache.org/jira/browse/HADOOP-16570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16570: Parent: HADOOP-15620 Issue Type: Sub-task (was: Bug) > S3A committers leak threads on job/task commit > -- > > Key: HADOOP-16570 > URL: https://issues.apache.org/jira/browse/HADOOP-16570 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0, 3.1.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > The fixed size ThreadPool created in AbstractS3ACommitter doesn't get cleaned > up at EOL; as a result you leak the no. of threads set in > "fs.s3a.committer.threads" > Not visible in MR/distcp jobs, but ultimately causes OOM on Spark -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-16520) race condition in DDB table init and waiting threads
[ https://issues.apache.org/jira/browse/HADOOP-16520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-16520 started by Gabor Bota. --- > race condition in DDB table init and waiting threads > > > Key: HADOOP-16520 > URL: https://issues.apache.org/jira/browse/HADOOP-16520 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > > s3guard threads waiting for table creation completion can be scheduled before > the creating thread, look for the version marker and then fail. > window will be sleep times in AWS SDK Table.waitForActive(); -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16520) race condition in DDB table init and waiting threads
[ https://issues.apache.org/jira/browse/HADOOP-16520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota reassigned HADOOP-16520: --- Assignee: Gabor Bota > race condition in DDB table init and waiting threads > > > Key: HADOOP-16520 > URL: https://issues.apache.org/jira/browse/HADOOP-16520 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > > s3guard threads waiting for table creation completion can be scheduled before > the creating thread, look for the version marker and then fail. > window will be sleep times in AWS SDK Table.waitForActive(); -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-16568) S3A FullCredentialsTokenBinding fails if local credentials are unset
[ https://issues.apache.org/jira/browse/HADOOP-16568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-16568 started by Steve Loughran. --- > S3A FullCredentialsTokenBinding fails if local credentials are unset > > > Key: HADOOP-16568 > URL: https://issues.apache.org/jira/browse/HADOOP-16568 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > Not sure how this slipped by the automated tests, but it is happening on my > CLI. > # FullCredentialsTokenBinding fails on startup if there are now AWS keys in > the auth chain > # because it tries to load them in serviceStart, not deployUnbonded -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16565) Fix "com.amazonaws.SdkClientException: Unable to find a region via the region provider chain."
[
https://issues.apache.org/jira/browse/HADOOP-16565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929313#comment-16929313
]
Steve Loughran commented on HADOOP-16565:
-
maybe you always need to set a region, and I'd not noticed. In which case we
should tune the defaults and document the error message
> Fix "com.amazonaws.SdkClientException: Unable to find a region via the region
> provider chain."
> --
>
> Key: HADOOP-16565
> URL: https://issues.apache.org/jira/browse/HADOOP-16565
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> The error found during testing in the following tests:
> {noformat}
> [ERROR] ITestS3ATemporaryCredentials.testInvalidSTSBinding:257 ? SdkClient
> Unable to f...
> [ERROR] ITestS3ATemporaryCredentials.testSTS:130 ? SdkClient Unable to find
> a region v...
> [ERROR]
> ITestS3ATemporaryCredentials.testSessionRequestExceptionTranslation:441->lambda$testSessionRequestExceptionTranslation$5:442
> ? SdkClient
> [ERROR] ITestS3ATemporaryCredentials.testSessionTokenExpiry:222 ? SdkClient
> Unable to ...
> [ERROR] ITestS3ATemporaryCredentials.testSessionTokenPropagation:193 ?
> SdkClient Unabl...
> [ERROR] ITestDelegatedMRJob.testJobSubmissionCollectsTokens:286 ? SdkClient
> Unable to ...
> [ERROR] ITestSessionDelegationInFileystem.testAddTokensFromFileSystem:235 ?
> SdkClient ...
> [ERROR]
> ITestSessionDelegationInFileystem.testCanRetrieveTokenFromCurrentUserCreds:260->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR]
> ITestSessionDelegationInFileystem.testDTCredentialProviderFromCurrentUserCreds:278->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR]
> ITestSessionDelegationInFileystem.testDelegatedFileSystem:308->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR]
> ITestSessionDelegationInFileystem.testDelegationBindingMismatch1:432->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR] ITestSessionDelegationInFileystem.testFileSystemBoundToCreator:681
> ? SdkClient
> [ERROR] ITestSessionDelegationInFileystem.testGetDTfromFileSystem:212 ?
> SdkClient Unab...
> [ERROR]
> ITestSessionDelegationInFileystem.testHDFSFetchDTCommand:606->lambda$testHDFSFetchDTCommand$3:607
> ? SdkClient
> [ERROR] ITestSessionDelegationInFileystem.testYarnCredentialPickup:576 ?
> SdkClient Una...
> [ERROR] ITestSessionDelegationTokens.testCreateAndUseDT:176 ? SdkClient
> Unable to find...
> [ERROR] ITestSessionDelegationTokens.testSaveLoadTokens:121 ? SdkClient
> Unable to find...
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1432: HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1
hadoop-yetus commented on issue #1432: HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1 URL: https://github.com/apache/hadoop/pull/1432#issuecomment-531290077 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 1160 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 65 | Maven dependency ordering for branch | | +1 | mvninstall | 1031 | trunk passed | | +1 | compile | 973 | trunk passed | | +1 | checkstyle | 150 | trunk passed | | +1 | mvnsite | 350 | trunk passed | | +1 | shadedclient | 1236 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 322 | trunk passed | | 0 | spotbugs | 34 | Used deprecated FindBugs config; considering switching to SpotBugs. | | 0 | findbugs | 33 | branch/hadoop-project no findbugs output file (findbugsXml.xml) | | 0 | findbugs | 34 | branch/hadoop-client-modules/hadoop-client-runtime no findbugs output file (findbugsXml.xml) | ||| _ Patch Compile Tests _ | | 0 | mvndep | 29 | Maven dependency ordering for patch | | +1 | mvninstall | 334 | the patch passed | | +1 | compile | 955 | the patch passed | | -1 | javac | 955 | root generated 396 new + 1466 unchanged - 0 fixed = 1862 total (was 1466) | | -0 | checkstyle | 142 | root: The patch generated 2 new + 35 unchanged - 1 fixed = 37 total (was 36) | | +1 | mvnsite | 350 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 4 | The patch has no ill-formed XML file. | | -1 | shadedclient | 61 | patch has errors when building and testing our client artifacts. | | +1 | javadoc | 358 | the patch passed | | 0 | findbugs | 31 | hadoop-project has no data from findbugs | | 0 | findbugs | 35 | hadoop-client-modules/hadoop-client-runtime has no data from findbugs | ||| _ Other Tests _ | | +1 | unit | 34 | hadoop-project in the patch passed. | | -1 | unit | 5158 | hadoop-hdfs in the patch failed. | | -1 | unit | 1454 | hadoop-hdfs-rbf in the patch failed. | | +1 | unit | 75 | hadoop-yarn-api in the patch passed. | | +1 | unit | 254 | hadoop-yarn-common in the patch passed. | | +1 | unit | 51 | hadoop-fs2img in the patch passed. | | +1 | unit | 42 | hadoop-client-runtime in the patch passed. | | +1 | asflicense | 63 | The patch does not generate ASF License warnings. | | | | 15745 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR | | | hadoop.hdfs.server.namenode.TestRedudantBlocks | | | hadoop.hdfs.server.datanode.TestLargeBlockReport | | | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks | | | hadoop.hdfs.server.federation.router.TestRouterWithSecureStartup | | | hadoop.hdfs.server.federation.security.TestRouterHttpDelegationToken | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1432/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1432 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle | | uname | Linux e8114dd0f88d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 39e82ac | | Default Java | 1.8.0_222 | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1432/5/artifact/out/diff-compile-javac-root.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1432/5/artifact/out/diff-checkstyle-root.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1432/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1432/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1432/5/testReport/ | | Max. process+thread count | 4113 (vs. ulimit of 5500) | | modules | C: hadoop-project hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-tools/hadoop-fs2img
[GitHub] [hadoop] hadoop-yetus commented on issue #1402: HADOOP-16547. make sure that s3guard prune sets up the FS
hadoop-yetus commented on issue #1402: HADOOP-16547. make sure that s3guard prune sets up the FS URL: https://github.com/apache/hadoop/pull/1402#issuecomment-531284055 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 1142 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1157 | trunk passed | | +1 | compile | 31 | trunk passed | | +1 | checkstyle | 24 | trunk passed | | +1 | mvnsite | 36 | trunk passed | | +1 | shadedclient | 784 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 25 | trunk passed | | 0 | spotbugs | 59 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 57 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 33 | the patch passed | | +1 | compile | 27 | the patch passed | | +1 | javac | 27 | the patch passed | | +1 | checkstyle | 18 | the patch passed | | +1 | mvnsite | 31 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 830 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 22 | the patch passed | | +1 | findbugs | 60 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 72 | hadoop-aws in the patch passed. | | +1 | asflicense | 29 | The patch does not generate ASF License warnings. | | | | 4473 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1402/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1402 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 16b5370caf95 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 95010a4 | | Default Java | 1.8.0_222 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1402/4/testReport/ | | Max. process+thread count | 350 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1402/4/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1438: HDDS-2126. Ozone 0.4.1 branch build issue.
hadoop-yetus commented on issue #1438: HDDS-2126. Ozone 0.4.1 branch build issue. URL: https://github.com/apache/hadoop/pull/1438#issuecomment-531283615 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 47 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ ozone-0.4.1 Compile Tests _ | | -1 | mvninstall | 362 | hadoop-ozone in ozone-0.4.1 failed. | | -1 | compile | 243 | hadoop-ozone in ozone-0.4.1 failed. | | -0 | checkstyle | 43 | The patch fails to run checkstyle in hadoop-ozone | | +1 | mvnsite | 0 | ozone-0.4.1 passed | | +1 | shadedclient | 873 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 91 | hadoop-ozone in ozone-0.4.1 failed. | | 0 | spotbugs | 201 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 362 | hadoop-ozone in ozone-0.4.1 failed. | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 542 | the patch passed | | +1 | compile | 390 | the patch passed | | -1 | javac | 286 | hadoop-ozone generated 2 new + 6 unchanged - 2 fixed = 8 total (was 8) | | +1 | checkstyle | 88 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 664 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 176 | the patch passed | | +1 | findbugs | 627 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 208 | hadoop-hdds in the patch failed. | | -1 | unit | 2008 | hadoop-ozone in the patch failed. | | +1 | asflicense | 59 | The patch does not generate ASF License warnings. | | | | 7509 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.ozoneimpl.TestOzoneContainer | | | hadoop.ozone.container.keyvalue.TestKeyValueContainer | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1438/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1438 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux a048c2b1e16d 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | ozone-0.4.1 / aaee64c | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1438/1/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1438/1/artifact/out/branch-compile-hadoop-ozone.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1438/1/artifact/out//home/jenkins/jenkins-slave/workspace/hadoop-multibranch_PR-1438/out/maven-branch-checkstyle-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1438/1/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1438/1/artifact/out/branch-findbugs-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1438/1/artifact/out/diff-compile-javac-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1438/1/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1438/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1438/1/testReport/ | | Max. process+thread count | 5036 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/container-service U: hadoop-hdds/container-service | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1438/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git
[jira] [Commented] (HADOOP-16569) improve TLS performance for NativeAzureFileSystem
[
https://issues.apache.org/jira/browse/HADOOP-16569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929292#comment-16929292
]
Sahil Takiar commented on HADOOP-16569:
---
Hey [~Steven Rand], thanks for tagging me. Yeah, HADOOP-16371 would help here.
I recently updated the PR for HADOOP-16371 -
https://github.com/apache/hadoop/pull/970 and am working on running some
additional testing, but review comments are welcome.
> improve TLS performance for NativeAzureFileSystem
> -
>
> Key: HADOOP-16569
> URL: https://issues.apache.org/jira/browse/HADOOP-16569
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Steven Rand
>Priority: Major
>
> Several tickets have already discussed the performance issues of GCM ciphers
> on jdk8 in relation to cloud connectors:
> * HADOOP-16050, HADOOP-16371 (s3a)
> * HADOOP-15669 (abfs)
> * HADOOP-15965 (adls)
> However, we haven't gotten around to fixing this for Azure Blob Store yet
> (the wasbs scheme).
> It would be helpful to reuse the {{SSLSocketFactoryEx}} class that was added
> in HADOOP-15669. The {{getInstrumentedContext}} method in
> {{AzureNativeFileSystemStore}} can already mutate the connection to the blob
> store and inject that socket factory, so it's not a hard change to make.
> This isn't necessarily blocked on HADOOP-16371, but it would be helpful if
> we merged that PR first, since it moves the custom {{SSLSocketFactory}} that
> uses wildfly-openssl out of the code that's specific to ABFS and into a place
> where the Azure Blob Store code can access it.
> Tagging [~stakiar] since you've been working on this for the other cloud
> connectors
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16565) Fix "com.amazonaws.SdkClientException: Unable to find a region via the region provider chain."
[
https://issues.apache.org/jira/browse/HADOOP-16565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota resolved HADOOP-16565.
-
Resolution: Workaround
> Fix "com.amazonaws.SdkClientException: Unable to find a region via the region
> provider chain."
> --
>
> Key: HADOOP-16565
> URL: https://issues.apache.org/jira/browse/HADOOP-16565
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> The error found during testing in the following tests:
> {noformat}
> [ERROR] ITestS3ATemporaryCredentials.testInvalidSTSBinding:257 ? SdkClient
> Unable to f...
> [ERROR] ITestS3ATemporaryCredentials.testSTS:130 ? SdkClient Unable to find
> a region v...
> [ERROR]
> ITestS3ATemporaryCredentials.testSessionRequestExceptionTranslation:441->lambda$testSessionRequestExceptionTranslation$5:442
> ? SdkClient
> [ERROR] ITestS3ATemporaryCredentials.testSessionTokenExpiry:222 ? SdkClient
> Unable to ...
> [ERROR] ITestS3ATemporaryCredentials.testSessionTokenPropagation:193 ?
> SdkClient Unabl...
> [ERROR] ITestDelegatedMRJob.testJobSubmissionCollectsTokens:286 ? SdkClient
> Unable to ...
> [ERROR] ITestSessionDelegationInFileystem.testAddTokensFromFileSystem:235 ?
> SdkClient ...
> [ERROR]
> ITestSessionDelegationInFileystem.testCanRetrieveTokenFromCurrentUserCreds:260->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR]
> ITestSessionDelegationInFileystem.testDTCredentialProviderFromCurrentUserCreds:278->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR]
> ITestSessionDelegationInFileystem.testDelegatedFileSystem:308->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR]
> ITestSessionDelegationInFileystem.testDelegationBindingMismatch1:432->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR] ITestSessionDelegationInFileystem.testFileSystemBoundToCreator:681
> ? SdkClient
> [ERROR] ITestSessionDelegationInFileystem.testGetDTfromFileSystem:212 ?
> SdkClient Unab...
> [ERROR]
> ITestSessionDelegationInFileystem.testHDFSFetchDTCommand:606->lambda$testHDFSFetchDTCommand$3:607
> ? SdkClient
> [ERROR] ITestSessionDelegationInFileystem.testYarnCredentialPickup:576 ?
> SdkClient Una...
> [ERROR] ITestSessionDelegationTokens.testCreateAndUseDT:176 ? SdkClient
> Unable to find...
> [ERROR] ITestSessionDelegationTokens.testSaveLoadTokens:121 ? SdkClient
> Unable to find...
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16565) Fix "com.amazonaws.SdkClientException: Unable to find a region via the region provider chain."
[
https://issues.apache.org/jira/browse/HADOOP-16565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929284#comment-16929284
]
Gabor Bota commented on HADOOP-16565:
-
It worked with the following STS config:
{noformat}
test.sts.endpoint
Specific endpoint to use for STS requests.
sts.amazonaws.com
sts.london.endpoint
sts.eu-west-2.amazonaws.com
sts.london.region
eu-west-2
sts.ireland.endpoint
sts.eu-west-1.amazonaws.com
fs.s3a.assumed.role.sts.endpoint
${sts.london.endpoint}
fs.s3a.assumed.role.sts.endpoint.region
${sts.london.region}
{noformat}
> Fix "com.amazonaws.SdkClientException: Unable to find a region via the region
> provider chain."
> --
>
> Key: HADOOP-16565
> URL: https://issues.apache.org/jira/browse/HADOOP-16565
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> The error found during testing in the following tests:
> {noformat}
> [ERROR] ITestS3ATemporaryCredentials.testInvalidSTSBinding:257 ? SdkClient
> Unable to f...
> [ERROR] ITestS3ATemporaryCredentials.testSTS:130 ? SdkClient Unable to find
> a region v...
> [ERROR]
> ITestS3ATemporaryCredentials.testSessionRequestExceptionTranslation:441->lambda$testSessionRequestExceptionTranslation$5:442
> ? SdkClient
> [ERROR] ITestS3ATemporaryCredentials.testSessionTokenExpiry:222 ? SdkClient
> Unable to ...
> [ERROR] ITestS3ATemporaryCredentials.testSessionTokenPropagation:193 ?
> SdkClient Unabl...
> [ERROR] ITestDelegatedMRJob.testJobSubmissionCollectsTokens:286 ? SdkClient
> Unable to ...
> [ERROR] ITestSessionDelegationInFileystem.testAddTokensFromFileSystem:235 ?
> SdkClient ...
> [ERROR]
> ITestSessionDelegationInFileystem.testCanRetrieveTokenFromCurrentUserCreds:260->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR]
> ITestSessionDelegationInFileystem.testDTCredentialProviderFromCurrentUserCreds:278->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR]
> ITestSessionDelegationInFileystem.testDelegatedFileSystem:308->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR]
> ITestSessionDelegationInFileystem.testDelegationBindingMismatch1:432->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR] ITestSessionDelegationInFileystem.testFileSystemBoundToCreator:681
> ? SdkClient
> [ERROR] ITestSessionDelegationInFileystem.testGetDTfromFileSystem:212 ?
> SdkClient Unab...
> [ERROR]
> ITestSessionDelegationInFileystem.testHDFSFetchDTCommand:606->lambda$testHDFSFetchDTCommand$3:607
> ? SdkClient
> [ERROR] ITestSessionDelegationInFileystem.testYarnCredentialPickup:576 ?
> SdkClient Una...
> [ERROR] ITestSessionDelegationTokens.testCreateAndUseDT:176 ? SdkClient
> Unable to find...
> [ERROR] ITestSessionDelegationTokens.testSaveLoadTokens:121 ? SdkClient
> Unable to find...
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16421) ITestS3GuardOutOfBandOperations.deleteAfterTombstoneExpiryOobCreate failure
[
https://issues.apache.org/jira/browse/HADOOP-16421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-16421.
-
Resolution: Cannot Reproduce
> ITestS3GuardOutOfBandOperations.deleteAfterTombstoneExpiryOobCreate failure
> ---
>
> Key: HADOOP-16421
> URL: https://issues.apache.org/jira/browse/HADOOP-16421
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.0
> Environment: AWS Ireland with a versioned object store set to delete
> old entries after 24h.
>Reporter: Steve Loughran
>Priority: Minor
>
> Saw a failure of
> ITestS3GuardOutOfBandOperations.deleteAfterTombstoneExpiryOobCreate
> {code}
> java.lang.AssertionError: This file should throw FNFE when reading through
> the raw fs, and the guarded fs deleted the file.:
> {code}
> Hypothesis: because the store is versioned, the old file was still there.
> Doesn't explain why I've not seen this before tho'
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16429) DynamoDBMetaStore deleteSubtree to delete leaf nodes first
[
https://issues.apache.org/jira/browse/HADOOP-16429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-16429.
-
Resolution: Done
seems to be done; as well as the deleteSubtree doing leaf nodes first,
HADOOP-16430 is moving to a list of all children and delete, which is now
linear rather than treewalk: leaf nodes are all you get
> DynamoDBMetaStore deleteSubtree to delete leaf nodes first
> --
>
> Key: HADOOP-16429
> URL: https://issues.apache.org/jira/browse/HADOOP-16429
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Priority: Major
>
> in {{deleteSubtree(path)}}, the DynamoDB metastore walks down the tree,
> returning elements to delete. But it will delete parent entries before
> children, so if an operation fails partway through, there will be orphans
> Better: DescendantsIterator to return all the leaf nodes before their parents
> so the deletion is done bottom up
> Also: push the deletions off into their own async queue/pool so that they
> don't become the bottleneck on the process
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16392) S3Guard Diff tool to list+ compare the etag and version fields
[ https://issues.apache.org/jira/browse/HADOOP-16392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-16392. - Resolution: Duplicate > S3Guard Diff tool to list+ compare the etag and version fields > -- > > Key: HADOOP-16392 > URL: https://issues.apache.org/jira/browse/HADOOP-16392 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Priority: Minor > > now that S3Guard supports etags and version IDs, the diff command should list > and compare them, -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16326) S3Guard: Remove LocalMetadataStore
[ https://issues.apache.org/jira/browse/HADOOP-16326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-16326. - Resolution: Won't Fix > S3Guard: Remove LocalMetadataStore > -- > > Key: HADOOP-16326 > URL: https://issues.apache.org/jira/browse/HADOOP-16326 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Priority: Minor > > We use LocalMetadataStore MetadataStore implementation in S3Guard only for > testing. > Inside it uses Guava's cache for storing metadatas. We try to mimic how > dynamo should work under the hood, but with every new feature or API > modification what we do on MetadataStore interface level it gets more and > more complicated to implement the same feature with different behavior. > I want to start a debate on why we need to remove that, or why we want to > keep it. > I could rant about why is it annoying to have this implementation when we > need to get some things right to work with dynamo, and then do a totally > different set of modification in the LocalMetadata to get the same outcome, > and also add more tests just for the thing we use for testing. > There are also areas in our ever-growing testing matrix that would need more > attention instead of fixing tests for our test implementation. But on the > other hand, it is good that we have another impl for the API which we can use > for drafting new ideas. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16274) transient failure of ITestS3GuardToolDynamoDB.testDestroyUnknownTable
[
https://issues.apache.org/jira/browse/HADOOP-16274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-16274.
-
Resolution: Cannot Reproduce
> transient failure of ITestS3GuardToolDynamoDB.testDestroyUnknownTable
> -
>
> Key: HADOOP-16274
> URL: https://issues.apache.org/jira/browse/HADOOP-16274
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Priority: Minor
>
> Experienced a transient failure of a test
> {code}
> [ERROR]
> testDestroyUnknownTable(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB)
> Time elapsed: 143.671 s <<< ERROR!
> java.lang.IllegalArgumentException: Table ireland-team is not deleted.
> {code}
> * The test run blocked for a while; I'd assumed network problems, but maybe
> it was retrying
> * verified on aWS console that the table was gone
> * Not surfaced on reruns
> I'm assuming this was transient, but anything going near creating tables runs
> a risk of creating bills. We need to move to on-demand table creation as soon
> as we upgrade the SDK
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16280) S3Guard: Retry failed read with backoff in Authoritative mode when file can be opened
[
https://issues.apache.org/jira/browse/HADOOP-16280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-16280.
-
Fix Version/s: 3.3.0
Resolution: Duplicate
HADOOP-16490 added exactly this retry logic
> S3Guard: Retry failed read with backoff in Authoritative mode when file can
> be opened
> -
>
> Key: HADOOP-16280
> URL: https://issues.apache.org/jira/browse/HADOOP-16280
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Gabor Bota
>Priority: Major
> Fix For: 3.3.0
>
>
> When using S3Guard in authoritative mode a file can be reported from AWS S3
> that's missing like it is described in the following exception:
> {noformat}
> java.io.FileNotFoundException: re-open
> s3a://cloudera-dev-gabor-ireland/test/TMCDOR-021df1ad-633f-47b8-97f5-6cd93f0b82d0
> at 0 on
> s3a://cloudera-dev-gabor-ireland/test/TMCDOR-021df1ad-633f-47b8-97f5-6cd93f0b82d0:
>
> com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not
> exist. (Service: Amazon S3; Status Code: 404; Error
> Code: NoSuchKey; Request ID: E1FF9EA9B5DBBD7E; S3 Extended Request ID:
> NzNIL4+dyA89WTnfbcwuYQK+hCfx51TfavwgC3oEvQI0IQ9M/zAspbXOfBIis8/nTolc4tRB9ik=),
> S3 Extended Request ID:
> NzNIL4+dyA89WTnfbcwuYQK+hCfx51TfavwgC3oEvQI0IQ9M/zAspbXOfBIis8/nTolc4tRB9ik=:NoSuchKey
> {noformat}
> But the metadata in S3Guard (e.g dynamo db) is there, so it can be opened.
> The operation will not fail when it's opened, it will fail when we try to
> read it, so the call
> {noformat}
> FSDataInputStream is = guardedFs.open(testFilePath);{noformat}
> won't fail, but the next call
> {noformat}
> byte[] firstRead = new byte[text.length()];
> is.read(firstRead, 0, firstRead.length);
> {noformat}
> will fail with the exception message like what's above.
> Once Authoritative mode is on, we assume that there is no out of band
> operation, so the file will appear eventually. We should re-try in this case.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16280) S3Guard: Retry failed read with backoff in Authoritative mode when file can be opened
[
https://issues.apache.org/jira/browse/HADOOP-16280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran reassigned HADOOP-16280:
---
Assignee: Steve Loughran
> S3Guard: Retry failed read with backoff in Authoritative mode when file can
> be opened
> -
>
> Key: HADOOP-16280
> URL: https://issues.apache.org/jira/browse/HADOOP-16280
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Gabor Bota
>Assignee: Steve Loughran
>Priority: Major
> Fix For: 3.3.0
>
>
> When using S3Guard in authoritative mode a file can be reported from AWS S3
> that's missing like it is described in the following exception:
> {noformat}
> java.io.FileNotFoundException: re-open
> s3a://cloudera-dev-gabor-ireland/test/TMCDOR-021df1ad-633f-47b8-97f5-6cd93f0b82d0
> at 0 on
> s3a://cloudera-dev-gabor-ireland/test/TMCDOR-021df1ad-633f-47b8-97f5-6cd93f0b82d0:
>
> com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not
> exist. (Service: Amazon S3; Status Code: 404; Error
> Code: NoSuchKey; Request ID: E1FF9EA9B5DBBD7E; S3 Extended Request ID:
> NzNIL4+dyA89WTnfbcwuYQK+hCfx51TfavwgC3oEvQI0IQ9M/zAspbXOfBIis8/nTolc4tRB9ik=),
> S3 Extended Request ID:
> NzNIL4+dyA89WTnfbcwuYQK+hCfx51TfavwgC3oEvQI0IQ9M/zAspbXOfBIis8/nTolc4tRB9ik=:NoSuchKey
> {noformat}
> But the metadata in S3Guard (e.g dynamo db) is there, so it can be opened.
> The operation will not fail when it's opened, it will fail when we try to
> read it, so the call
> {noformat}
> FSDataInputStream is = guardedFs.open(testFilePath);{noformat}
> won't fail, but the next call
> {noformat}
> byte[] firstRead = new byte[text.length()];
> is.read(firstRead, 0, firstRead.length);
> {noformat}
> will fail with the exception message like what's above.
> Once Authoritative mode is on, we assume that there is no out of band
> operation, so the file will appear eventually. We should re-try in this case.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15800) ITestS3GuardListConsistency#testConsistentListAfterDelete fails when running with dynamo
[
https://issues.apache.org/jira/browse/HADOOP-15800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-15800.
-
Resolution: Cannot Reproduce
> ITestS3GuardListConsistency#testConsistentListAfterDelete fails when running
> with dynamo
>
>
> Key: HADOOP-15800
> URL: https://issues.apache.org/jira/browse/HADOOP-15800
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.1
>Reporter: Gabor Bota
>Priority: Major
>
> I've seen a new failure when running verify for HADOOP-15621. First I thought
> it was my new patch, but it happens on trunk. This is a major issue, it could
> be because of implementation issue in dynamo.
> {noformat}
> [ERROR]
> testConsistentListAfterDelete(org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency)
> Time elapsed: 2.212 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testConsistentListAfterDelete(ITestS3GuardListConsistency.java:205)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74){noformat}
> Tested against {{us-west-1}}, I was on the same region.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16101) Use lighter-weight alternatives to innerGetFileStatus where possible
[ https://issues.apache.org/jira/browse/HADOOP-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16101: Parent Issue: HADOOP-15620 (was: HADOOP-15619) > Use lighter-weight alternatives to innerGetFileStatus where possible > > > Key: HADOOP-16101 > URL: https://issues.apache.org/jira/browse/HADOOP-16101 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sean Mackrory >Priority: Major > > Discussion in HADOOP-15999 highlighted the heaviness of a full > innerGetFileStatus call, where many usages of it may need a lighter weight > fileExists, etc. Let's investigate usage of innerGetFileStatus and slim it > down where possible. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1440: HDDS-2114: Rename does not preserve non-explicitly created interim directories
hadoop-yetus commented on issue #1440: HDDS-2114: Rename does not preserve non-explicitly created interim directories URL: https://github.com/apache/hadoop/pull/1440#issuecomment-531270620 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 36 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | -1 | mvninstall | 31 | hadoop-ozone in trunk failed. | | -1 | compile | 19 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 64 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 896 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 13 | hadoop-hdds in trunk failed. | | -1 | javadoc | 13 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 159 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 22 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | -1 | mvninstall | 30 | hadoop-ozone in the patch failed. | | -1 | compile | 20 | hadoop-ozone in the patch failed. | | -1 | javac | 20 | hadoop-ozone in the patch failed. | | -0 | checkstyle | 26 | hadoop-ozone: The patch generated 2 new + 121 unchanged - 0 fixed = 123 total (was 121) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 716 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 13 | hadoop-hdds in the patch failed. | | -1 | javadoc | 13 | hadoop-ozone in the patch failed. | | -1 | findbugs | 21 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 137 | hadoop-hdds in the patch failed. | | -1 | unit | 24 | hadoop-ozone in the patch failed. | | +1 | asflicense | 28 | The patch does not generate ASF License warnings. | | | | 2860 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.keyvalue.TestKeyValueContainer | | | hadoop.ozone.container.ozoneimpl.TestOzoneContainer | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1440 | | JIRA Issue | HDDS-2114 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux c3b727f210a2 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 39e82ac | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/patch-compile-hadoop-ozone.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/diff-checkstyle-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/patch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/patch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/patch-findbugs-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/testReport/ | | Max. process+thread count |
[jira] [Updated] (HADOOP-16568) S3A FullCredentialsTokenBinding fails if local credentials are unset
[ https://issues.apache.org/jira/browse/HADOOP-16568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16568: Parent Issue: HADOOP-15620 (was: HADOOP-15619) > S3A FullCredentialsTokenBinding fails if local credentials are unset > > > Key: HADOOP-16568 > URL: https://issues.apache.org/jira/browse/HADOOP-16568 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > Not sure how this slipped by the automated tests, but it is happening on my > CLI. > # FullCredentialsTokenBinding fails on startup if there are now AWS keys in > the auth chain > # because it tries to load them in serviceStart, not deployUnbonded -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1404: HDFS-13660 Copy file till the source file length during distcp
steveloughran commented on a change in pull request #1404: HDFS-13660 Copy file
till the source file length during distcp
URL: https://github.com/apache/hadoop/pull/1404#discussion_r324232111
##
File path:
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/util/TestDistCpUtils.java
##
@@ -1224,22 +1225,18 @@ public void testCompareFileLengthsAndChecksums()
throws IOException {
Path dstWithLen0 = new Path(base + "dstLen0");
fs.create(srcWithLen0).close();
fs.create(dstWithLen0).close();
-DistCpUtils.compareFileLengthsAndChecksums(fs, srcWithLen0,
-null, fs, dstWithLen0, false);
+DistCpUtils.compareFileLengthsAndChecksums(0, fs, srcWithLen0,
+null, fs, dstWithLen0, false, 0);
// different lengths comparison
Path srcWithLen1 = new Path(base + "srcLen1");
Path dstWithLen2 = new Path(base + "dstLen2");
DFSTestUtil.createFile(fs, srcWithLen1, 1, replFactor, srcSeed);
DFSTestUtil.createFile(fs, dstWithLen2, 2, replFactor, srcSeed);
-try {
- DistCpUtils.compareFileLengthsAndChecksums(fs, srcWithLen1,
- null, fs, dstWithLen2, false);
- Assert.fail("Expected different lengths comparison to fail!");
-} catch (IOException e) {
- GenericTestUtils.assertExceptionContains(
- "Mismatch in length", e);
-}
+
+intercept(IOException.class, DistCpConstants.LENGTH_MISMATCH_ERROR_MSG,
Review comment:
this is good, thanks for doing this
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1404: HDFS-13660 Copy file till the source file length during distcp
steveloughran commented on a change in pull request #1404: HDFS-13660 Copy file till the source file length during distcp URL: https://github.com/apache/hadoop/pull/1404#discussion_r324231824 ## File path: hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestRetriableFileCopyCommand.java ## @@ -24,6 +24,7 @@ import org.apache.hadoop.mapreduce.*; import org.apache.hadoop.tools.CopyListingFileStatus; import org.apache.hadoop.tools.mapred.CopyMapper.FileAction; +import org.junit.Assert; Review comment: stick a newline in above this This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1404: HDFS-13660 Copy file till the source file length during distcp
steveloughran commented on a change in pull request #1404: HDFS-13660 Copy file
till the source file length during distcp
URL: https://github.com/apache/hadoop/pull/1404#discussion_r324231592
##
File path:
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyMapper.java
##
@@ -444,6 +449,54 @@ private void testCopyingExistingFiles(FileSystem fs,
CopyMapper copyMapper,
}
}
+ @Test
+ public void testCopyWhileAppend() throws Exception {
+deleteState();
+mkdirs(SOURCE_PATH + "/1");
+touchFile(SOURCE_PATH + "/1/3");
+CopyMapper copyMapper = new CopyMapper();
+StubContext stubContext = new StubContext(getConfiguration(), null, 0);
+Mapper.Context context =
+stubContext.getContext();
+copyMapper.setup(context);
+final Path path = new Path(SOURCE_PATH + "/1/3");
+int manyBytes = 1;
+appendFile(path, manyBytes);
+ScheduledExecutorService scheduledExecutorService =
Executors.newSingleThreadScheduledExecutor();
+Runnable task = new Runnable() {
+ public void run() {
+try {
+ int maxAppendAttempts = 20;
+ int appendCount = 0;
+ while (appendCount < maxAppendAttempts) {
+appendFile(path, 1000);
+Thread.sleep(200);
+appendCount++;
+ }
+} catch (IOException | InterruptedException e) {
+ e.printStackTrace();
+}
+ }
+};
+scheduledExecutorService.schedule(task, 10, TimeUnit.MILLISECONDS);
+boolean isFileMismatchErrorPresent = false;
+try {
Review comment:
good point. But: why then do you catch some exceptions and discard them?
What other IOEs get raised which aren't errors?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1404: HDFS-13660 Copy file till the source file length during distcp
steveloughran commented on a change in pull request #1404: HDFS-13660 Copy file
till the source file length during distcp
URL: https://github.com/apache/hadoop/pull/1404#discussion_r324231054
##
File path:
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyMapper.java
##
@@ -444,6 +450,57 @@ private void testCopyingExistingFiles(FileSystem fs,
CopyMapper copyMapper,
}
}
+ @Test(timeout = 4)
+ public void testCopyWhileAppend() throws Exception {
+deleteState();
+mkdirs(SOURCE_PATH + "/1");
+touchFile(SOURCE_PATH + "/1/3");
+CopyMapper copyMapper = new CopyMapper();
+StubContext stubContext = new StubContext(getConfiguration(), null, 0);
+Mapper.Context context =
+stubContext.getContext();
+copyMapper.setup(context);
+final Path path = new Path(SOURCE_PATH + "/1/3");
+int manyBytes = 1;
+appendFile(path, manyBytes);
+ScheduledExecutorService scheduledExecutorService =
+Executors.newSingleThreadScheduledExecutor();
+Runnable task = new Runnable() {
+ public void run() {
+try {
+ int maxAppendAttempts = 20;
+ int appendCount = 0;
+ while (appendCount < maxAppendAttempts) {
+appendFile(path, 1000);
+Thread.sleep(200);
+appendCount++;
+ }
+} catch (IOException | InterruptedException e) {
+LOG.error("Exception encountered ", e);
+Assert.fail("Test failed: " + e.getMessage());
Review comment:
`throw new AssertionError(Test failed: " + e,e)`
Stack traces are too import to throw away
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1404: HDFS-13660 Copy file till the source file length during distcp
steveloughran commented on a change in pull request #1404: HDFS-13660 Copy file till the source file length during distcp URL: https://github.com/apache/hadoop/pull/1404#discussion_r324230455 ## File path: hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyMapper.java ## @@ -21,12 +21,18 @@ import java.io.DataOutputStream; import java.io.IOException; import java.io.OutputStream; +import java.io.PrintWriter; +import java.io.StringWriter; import java.security.PrivilegedAction; import java.util.ArrayList; import java.util.EnumSet; import java.util.List; import java.util.Random; +import java.util.concurrent.Executors; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.TimeUnit; +import org.apache.hadoop.test.LambdaTestUtils; Review comment: again, org.apache. entries together, in order. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1404: HDFS-13660 Copy file till the source file length during distcp
steveloughran commented on a change in pull request #1404: HDFS-13660 Copy file till the source file length during distcp URL: https://github.com/apache/hadoop/pull/1404#discussion_r324230187 ## File path: hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java ## @@ -20,6 +20,7 @@ import com.google.common.collect.Maps; +import org.apache.hadoop.tools.DistCpConstants; Review comment: can you move this to be in the org.apache bit of the imports This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] lokeshj1703 commented on issue #1430: HDDS-2117. ContainerStateMachine#writeStateMachineData times out.
lokeshj1703 commented on issue #1430: HDDS-2117. ContainerStateMachine#writeStateMachineData times out. URL: https://github.com/apache/hadoop/pull/1430#issuecomment-531260923 @bshashikant Thanks for updating the PR! The changes look good to me. +1. Pending jenkins. The yetus build seems to have an issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16547) s3guard prune command doesn't get AWS auth chain from FS
[
https://issues.apache.org/jira/browse/HADOOP-16547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929243#comment-16929243
]
Steve Loughran commented on HADOOP-16547:
-
with the test plan, I can verify that prune, set-capacity, destroy and init
fail to work with Delegation Token auth. with the patch, all of these *except
init* work. Init is special as because its initing the bucket, it doesn't want
a filesystem. I'm not worrying about this, as its a major admin command which
you wouldn't normally use through DTs.
Old code
{code}
~/P/R/fsck bin/hadoop s3guard prune -seconds 0 -tombstone
s3a://hwdev-steve-ireland-new/
java.nio.file.AccessDeniedException: hwdev-steve-ireland-new:
org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials
provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider
EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider :
com.amazonaws.SdkClientException: Unable to load AWS credentials from
environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY
(or AWS_SECRET_ACCESS_KEY))
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:200)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initTable(DynamoDBMetadataStore.java:1840)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:521)
at
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.initMetadataStore(S3GuardTool.java:318)
at
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$Prune.run(S3GuardTool.java:1072)
at
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:402)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:1767)
at
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.main(S3GuardTool.java:1776)
Caused by: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS
Credentials provided by TemporaryAWSCredentialsProvider
SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider
IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to
load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or
AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
at
org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:216)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1225)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:801)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:751)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:4279)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:4246)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeDescribeTable(AmazonDynamoDBClient.java:1905)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.describeTable(AmazonDynamoDBClient.java:1871)
at
com.amazonaws.services.dynamodbv2.document.Table.describe(Table.java:137)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initTable(DynamoDBMetadataStore.java:1775)
... 7 more
Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials
from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and
AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
at
com.amazonaws.auth.EnvironmentVariableCredentialsProvider.getCredentials(EnvironmentVariableCredentialsProvider.java:50)
at
org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:177)
... 22 more
2019-09-13 14:19:23,373 [main] INFO util.ExitUtil
(ExitUtil.java:terminate(210)) - Exiting with status -1:
java.nio.file.AccessDeniedException: hwdev-steve-ireland-new:
org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials
provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider
EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider :
com.amazonaws.SdkClientException: Unable to load AWS
[jira] [Commented] (HADOOP-16565) Fix "com.amazonaws.SdkClientException: Unable to find a region via the region provider chain."
[
https://issues.apache.org/jira/browse/HADOOP-16565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929242#comment-16929242
]
Gabor Bota commented on HADOOP-16565:
-
I don't have any STS endpoint besides
{noformat}
test.sts.endpoint
Specific endpoint to use for STS requests.
sts.amazonaws.com
{noformat}
so I haven't changed a single setting and I was not getting this error but now
I'm getting it.
> Fix "com.amazonaws.SdkClientException: Unable to find a region via the region
> provider chain."
> --
>
> Key: HADOOP-16565
> URL: https://issues.apache.org/jira/browse/HADOOP-16565
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> The error found during testing in the following tests:
> {noformat}
> [ERROR] ITestS3ATemporaryCredentials.testInvalidSTSBinding:257 ? SdkClient
> Unable to f...
> [ERROR] ITestS3ATemporaryCredentials.testSTS:130 ? SdkClient Unable to find
> a region v...
> [ERROR]
> ITestS3ATemporaryCredentials.testSessionRequestExceptionTranslation:441->lambda$testSessionRequestExceptionTranslation$5:442
> ? SdkClient
> [ERROR] ITestS3ATemporaryCredentials.testSessionTokenExpiry:222 ? SdkClient
> Unable to ...
> [ERROR] ITestS3ATemporaryCredentials.testSessionTokenPropagation:193 ?
> SdkClient Unabl...
> [ERROR] ITestDelegatedMRJob.testJobSubmissionCollectsTokens:286 ? SdkClient
> Unable to ...
> [ERROR] ITestSessionDelegationInFileystem.testAddTokensFromFileSystem:235 ?
> SdkClient ...
> [ERROR]
> ITestSessionDelegationInFileystem.testCanRetrieveTokenFromCurrentUserCreds:260->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR]
> ITestSessionDelegationInFileystem.testDTCredentialProviderFromCurrentUserCreds:278->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR]
> ITestSessionDelegationInFileystem.testDelegatedFileSystem:308->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR]
> ITestSessionDelegationInFileystem.testDelegationBindingMismatch1:432->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR] ITestSessionDelegationInFileystem.testFileSystemBoundToCreator:681
> ? SdkClient
> [ERROR] ITestSessionDelegationInFileystem.testGetDTfromFileSystem:212 ?
> SdkClient Unab...
> [ERROR]
> ITestSessionDelegationInFileystem.testHDFSFetchDTCommand:606->lambda$testHDFSFetchDTCommand$3:607
> ? SdkClient
> [ERROR] ITestSessionDelegationInFileystem.testYarnCredentialPickup:576 ?
> SdkClient Una...
> [ERROR] ITestSessionDelegationTokens.testCreateAndUseDT:176 ? SdkClient
> Unable to find...
> [ERROR] ITestSessionDelegationTokens.testSaveLoadTokens:121 ? SdkClient
> Unable to find...
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16547) s3guard prune command doesn't get AWS auth chain from FS
[ https://issues.apache.org/jira/browse/HADOOP-16547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929239#comment-16929239 ] Steve Loughran commented on HADOOP-16547: - * build a full Hadoop distro without the patch * Enable delegation tokens for a test bucket. * use fetchdt to collect a token for that bucket * set the HADOOP_TOKEN_FILE_LOCATION environment variable to point at the token file * unset the AWS credentials * set the ddb enabled, region and bucket names for all buckets, so ddb can try to init even when unbound to an FS. * execute all the s3guard cli operations; observe which fail * build a hadoop release *with* the patch, verify the failing operations now succeed > s3guard prune command doesn't get AWS auth chain from FS > > > Key: HADOOP-16547 > URL: https://issues.apache.org/jira/browse/HADOOP-16547 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > s3guard prune command doesn't get AWS auth chain from any FS, so it just > drives the DDB store from the conf settings. If S3A is set up to use > Delegation tokens then the DTs/custom AWS auth sequence is not picked up, so > you get an auth failure. > Fix: > # instantiate the FS before calling initMetadataStore > # review other commands to make sure problem isn't replicated -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1439: HDDS-2125. maven-javadoc-plugin.version is missing in pom.ozone.xml.
hadoop-yetus commented on issue #1439: HDDS-2125. maven-javadoc-plugin.version is missing in pom.ozone.xml. URL: https://github.com/apache/hadoop/pull/1439#issuecomment-531255241 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | ||| _ trunk Compile Tests _ | | +1 | shadedclient | 761 | branch has no errors when building and testing our client artifacts. | ||| _ Patch Compile Tests _ | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 1 | The patch has no ill-formed XML file. | | +1 | shadedclient | 770 | patch has no errors when building and testing our client artifacts. | ||| _ Other Tests _ | | +1 | asflicense | 31 | The patch does not generate ASF License warnings. | | | | 2785 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1439/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1439 | | Optional Tests | dupname asflicense xml | | uname | Linux 9d7b48b3a86d 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 39e82ac | | Max. process+thread count | 412 (vs. ulimit of 5500) | | modules | C: . U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1439/1/console | | versions | git=2.7.4 maven=3.3.9 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1402: HADOOP-16547. make sure that s3guard prune sets up the FS
steveloughran commented on issue #1402: HADOOP-16547. make sure that s3guard prune sets up the FS URL: https://github.com/apache/hadoop/pull/1402#issuecomment-531254101 manually verified this patch works through DT games, as well as ITest run. All good This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on issue #1377: HDDS-2057. Incorrect Default OM Port in Ozone FS URI Error Message. Contributed by Supratim Deka
anuengineer commented on issue #1377: HDDS-2057. Incorrect Default OM Port in Ozone FS URI Error Message. Contributed by Supratim Deka URL: https://github.com/apache/hadoop/pull/1377#issuecomment-531253622 @supratimdeka Thanks for the contribution. @arp7 , @bharatviswa504 Thanks for the reviews. I have committed this to the trunk branch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
