[GitHub] [hadoop] hadoop-yetus commented on pull request #4088: HDFS-16514. Reduce the failover sleep time if multiple namenode are c…
hadoop-yetus commented on pull request #4088: URL: https://github.com/apache/hadoop/pull/4088#issuecomment-1074823545 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 42s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 1s | | codespell was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | _ trunk Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for branch | | -1 :x: | mvninstall | 0m 25s | [/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4088/2/artifact/out/branch-mvninstall-root.txt) | root in trunk failed. | | -1 :x: | compile | 0m 25s | [/branch-compile-root-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4088/2/artifact/out/branch-compile-root-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt) | root in trunk failed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04. | | -1 :x: | compile | 0m 24s | [/branch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4088/2/artifact/out/branch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root in trunk failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. | | -0 :warning: | checkstyle | 0m 22s | [/buildtool-branch-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4088/2/artifact/out/buildtool-branch-checkstyle-root.txt) | The patch fails to run checkstyle in root | | -1 :x: | mvnsite | 0m 25s | [/branch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4088/2/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt) | hadoop-common in trunk failed. | | -1 :x: | mvnsite | 0m 23s | [/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4088/2/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-client.txt) | hadoop-hdfs-client in trunk failed. | | -1 :x: | javadoc | 0m 23s | [/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4088/2/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt) | hadoop-common in trunk failed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04. | | -1 :x: | javadoc | 0m 23s | [/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4088/2/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt) | hadoop-hdfs-client in trunk failed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04. | | -1 :x: | javadoc | 0m 23s | [/branch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4088/2/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-common in trunk failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. | | -1 :x: | javadoc | 0m 23s | [/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4088/2/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-hdfs-client in trunk failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. | | -1 :x: | spotbugs | 0m 23s | [/branch-spotbugs-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4088/2/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common.txt) | hadoop-common in trunk failed. | | -1 :x: | spotbugs | 0m 24s | [/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4088/2/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.txt) | hadoop-hdfs-client in trunk failed. | | +1 :green_heart:
[GitHub] [hadoop] hadoop-yetus commented on pull request #4074: HADOOP-18160 Avoid shading wildfly.openssl runtime dependency
hadoop-yetus commented on pull request #4074: URL: https://github.com/apache/hadoop/pull/4074#issuecomment-1074997394 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 1m 8s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 36m 47s | | trunk passed | | +1 :green_heart: | compile | 0m 18s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | compile | 0m 18s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | mvnsite | 0m 22s | | trunk passed | | +1 :green_heart: | javadoc | 0m 20s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 0m 19s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | shadedclient | 60m 55s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 16s | | the patch passed | | +1 :green_heart: | compile | 0m 12s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javac | 0m 12s | | the patch passed | | +1 :green_heart: | compile | 0m 12s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | javac | 0m 12s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | +1 :green_heart: | mvnsite | 0m 15s | | the patch passed | | +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 0m 12s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 0m 13s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | shadedclient | 23m 8s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 0m 15s | | hadoop-client-api in the patch passed. | | +1 :green_heart: | asflicense | 0m 31s | | The patch does not generate ASF License warnings. | | | | 91m 22s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4074/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4074 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml | | uname | Linux 4d1ed5004657 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8 19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 37f4364c44106f95854e8acc2af99382d2acb89c | | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4074/3/testReport/ | | Max. process+thread count | 521 (vs. ulimit of 5500) | | modules | C: hadoop-client-modules/hadoop-client-api U: hadoop-client-modules/hadoop-client-api | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4074/3/console | | versions | git=2.25.1 maven=3.6.3 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apac
[jira] [Work logged] (HADOOP-18160) `org.wildfly.openssl` should not be shaded by Hadoop build
[ https://issues.apache.org/jira/browse/HADOOP-18160?focusedWorklogId=745700&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745700 ] ASF GitHub Bot logged work on HADOOP-18160: --- Author: ASF GitHub Bot Created on: 22/Mar/22 10:28 Start Date: 22/Mar/22 10:28 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #4074: URL: https://github.com/apache/hadoop/pull/4074#issuecomment-1074997394 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 1m 8s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 36m 47s | | trunk passed | | +1 :green_heart: | compile | 0m 18s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | compile | 0m 18s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | mvnsite | 0m 22s | | trunk passed | | +1 :green_heart: | javadoc | 0m 20s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 0m 19s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | shadedclient | 60m 55s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 16s | | the patch passed | | +1 :green_heart: | compile | 0m 12s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javac | 0m 12s | | the patch passed | | +1 :green_heart: | compile | 0m 12s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | javac | 0m 12s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | +1 :green_heart: | mvnsite | 0m 15s | | the patch passed | | +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 0m 12s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 0m 13s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | shadedclient | 23m 8s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 0m 15s | | hadoop-client-api in the patch passed. | | +1 :green_heart: | asflicense | 0m 31s | | The patch does not generate ASF License warnings. | | | | 91m 22s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4074/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4074 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml | | uname | Linux 4d1ed5004657 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8 19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 37f4364c44106f95854e8acc2af99382d2acb89c | | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4074/3/testReport/ | | Max. process+thread count | 521 (vs. ulimit of 5500) | | modules | C: hadoop-client-modules/hadoop-client-api U: hadoop-client-modules/hadoop-client-api | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4074/3/console | | versions | git=2.25.1 maven=3.6.3 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. -- This is an automated m
[GitHub] [hadoop] hadoop-yetus commented on pull request #4069: HDFS-16457.Make fs.getspaceused.classname reconfigurable
hadoop-yetus commented on pull request #4069: URL: https://github.com/apache/hadoop/pull/4069#issuecomment-1075007002 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 1m 0s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 1s | | codespell was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 33m 45s | | trunk passed | | +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | checkstyle | 1m 2s | | trunk passed | | +1 :green_heart: | mvnsite | 1m 30s | | trunk passed | | +1 :green_heart: | javadoc | 1m 5s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 3m 17s | | trunk passed | | +1 :green_heart: | shadedclient | 22m 36s | | branch has no errors when building and testing our client artifacts. | | -0 :warning: | patch | 22m 55s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 19s | | the patch passed | | +1 :green_heart: | compile | 1m 20s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javac | 1m 20s | | the patch passed | | +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | javac | 1m 15s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | -0 :warning: | checkstyle | 0m 53s | [/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4069/7/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 205 unchanged - 0 fixed = 206 total (was 205) | | +1 :green_heart: | mvnsite | 1m 19s | | the patch passed | | +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 1m 24s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 3m 17s | | the patch passed | | +1 :green_heart: | shadedclient | 22m 27s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | -1 :x: | unit | 389m 56s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4069/7/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 0m 49s | | The patch does not generate ASF License warnings. | | | | 491m 9s | | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.namenode.ha.TestEditLogTailer | | | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes | | | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover | | | hadoop.hdfs.server.namenode.TestFileTruncate | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4069/7/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4069 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell | | uname | Linux 9313644d2743 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 68b4a871737b033bce5fcb207e6868d53b910632 | | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07
[GitHub] [hadoop] dannycjones commented on a change in pull request #4070: HADOOP-18154. S3A Authentication to support WebIdentity
dannycjones commented on a change in pull request #4070: URL: https://github.com/apache/hadoop/pull/4070#discussion_r832005639 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java ## @@ -142,6 +142,10 @@ private Constants() { public static final String ASSUMED_ROLE_CREDENTIALS_DEFAULT = SimpleAWSCredentialsProvider.NAME; + /** + * Absolute path to the web identity token file + */ + public static final String JWT_PATH = "fs.s3a.jwt.path"; Review comment: If its for OIDC / WebIdentity, can we change to something like `fs.s3a.oidc.jwt.path` / `fs.s3a.webidentity.jwt.path`? ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/OIDCTokenCredentialsProvider.java ## @@ -0,0 +1,105 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.s3a; + +import com.amazonaws.auth.AWSCredentials; +import com.amazonaws.auth.AWSCredentialsProvider; +import com.amazonaws.auth.WebIdentityTokenCredentialsProvider; + +import org.apache.commons.lang3.StringUtils; + +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.classification.InterfaceStability; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.security.ProviderUtils; + +import org.slf4j.Logger; + +import java.io.IOException; + +import static org.apache.hadoop.fs.s3a.Constants.*; + +/** + * Support OpenID Connect (OIDC) token for authenticating with AWS. + * + * Please note that users may reference this class name from configuration + * property fs.s3a.aws.credentials.provider. Therefore, changing the class name + * would be a backward-incompatible change. + * + * This credential provider must not fail in creation because that will + * break a chain of credential providers. + */ Review comment: Does this credential provider actually support more than just Open ID Connect - anything that vends an identity under a JWT? This provider is allowing users to configure Role ARN, JWT path, and session name for the [SDK WebIdentityTokenCredentialsProvider](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/WebIdentityTokenCredentialsProvider.html). Should we move to similar naming? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18154) S3A Authentication to support WebIdentity
[ https://issues.apache.org/jira/browse/HADOOP-18154?focusedWorklogId=745713&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745713 ] ASF GitHub Bot logged work on HADOOP-18154: --- Author: ASF GitHub Bot Created on: 22/Mar/22 10:59 Start Date: 22/Mar/22 10:59 Worklog Time Spent: 10m Work Description: dannycjones commented on a change in pull request #4070: URL: https://github.com/apache/hadoop/pull/4070#discussion_r832005639 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java ## @@ -142,6 +142,10 @@ private Constants() { public static final String ASSUMED_ROLE_CREDENTIALS_DEFAULT = SimpleAWSCredentialsProvider.NAME; + /** + * Absolute path to the web identity token file + */ + public static final String JWT_PATH = "fs.s3a.jwt.path"; Review comment: If its for OIDC / WebIdentity, can we change to something like `fs.s3a.oidc.jwt.path` / `fs.s3a.webidentity.jwt.path`? ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/OIDCTokenCredentialsProvider.java ## @@ -0,0 +1,105 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.s3a; + +import com.amazonaws.auth.AWSCredentials; +import com.amazonaws.auth.AWSCredentialsProvider; +import com.amazonaws.auth.WebIdentityTokenCredentialsProvider; + +import org.apache.commons.lang3.StringUtils; + +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.classification.InterfaceStability; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.security.ProviderUtils; + +import org.slf4j.Logger; + +import java.io.IOException; + +import static org.apache.hadoop.fs.s3a.Constants.*; + +/** + * Support OpenID Connect (OIDC) token for authenticating with AWS. + * + * Please note that users may reference this class name from configuration + * property fs.s3a.aws.credentials.provider. Therefore, changing the class name + * would be a backward-incompatible change. + * + * This credential provider must not fail in creation because that will + * break a chain of credential providers. + */ Review comment: Does this credential provider actually support more than just Open ID Connect - anything that vends an identity under a JWT? This provider is allowing users to configure Role ARN, JWT path, and session name for the [SDK WebIdentityTokenCredentialsProvider](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/WebIdentityTokenCredentialsProvider.html). Should we move to similar naming? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745713) Time Spent: 2h 20m (was: 2h 10m) > S3A Authentication to support WebIdentity > - > > Key: HADOOP-18154 > URL: https://issues.apache.org/jira/browse/HADOOP-18154 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 2.10.1 >Reporter: Ju Clarysse >Assignee: Ju Clarysse >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > We are using the latest version of > [delta-sharing|https://github.com/delta-io/delta-sharing] which takes > advantage of > [hadoop-aws|https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html] > (S3A) connector in [Hadoop release version > 2.10.1|https://github.com/apache/hadoop/tree/rel/release-2.10.1] to mount an > AWS S3 File System. In our particular setup, all services are operated in > Amazon Elastic Kubernetes Service (EKS) and need to comply to the AWS > security concept [IAM
[GitHub] [hadoop] ayushtkn commented on pull request #4089: HDFS-16515. Improve ec exception message
ayushtkn commented on pull request #4089: URL: https://github.com/apache/hadoop/pull/4089#issuecomment-1075030278 I think this message is propagated to end client and might be on CLI as well. IMO this would be an incompatible change, we should refrain from doing this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18167) Add metrics to track delegation token secret manager operations
[ https://issues.apache.org/jira/browse/HADOOP-18167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510424#comment-17510424 ] Steve Loughran commented on HADOOP-18167: - might be nice to have it implement IOStatistic and so collect per instance stats which can be aggregated/reported > Add metrics to track delegation token secret manager operations > --- > > Key: HADOOP-18167 > URL: https://issues.apache.org/jira/browse/HADOOP-18167 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Priority: Major > > New metrics to track operations that store, update and remove delegation > tokens in implementations of AbstractDelegationTokenSecretManager. This will > help evaluate the impact of using different secret managers and add > optimizations. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] GuoPhilipse commented on pull request #4089: HDFS-16515. Improve ec exception message
GuoPhilipse commented on pull request #4089: URL: https://github.com/apache/hadoop/pull/4089#issuecomment-1075078314 > > Yes,thanks @ayushtkn for your review, it shows for the end client on CLI output, I just think the propagated message not that clear for the usage of EC , i need to recheck the information on the website to confirm that only directory can be accepted or search the message in the source code. if we unset erasure coding policy for a file . the CLI out will show as `Cannot unset an erasure coding policy on a file `, which relatively tells end client more clearer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] zuston commented on pull request #4060: YARN-11084. Introduce new config to specify AM default node-label whe…
zuston commented on pull request #4060: URL: https://github.com/apache/hadoop/pull/4060#issuecomment-1075080288 All Done @9uapaw -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] dannycjones commented on a change in pull request #4070: HADOOP-18154. S3A Authentication to support WebIdentity
dannycjones commented on a change in pull request #4070: URL: https://github.com/apache/hadoop/pull/4070#discussion_r832005639 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java ## @@ -142,6 +142,10 @@ private Constants() { public static final String ASSUMED_ROLE_CREDENTIALS_DEFAULT = SimpleAWSCredentialsProvider.NAME; + /** + * Absolute path to the web identity token file + */ + public static final String JWT_PATH = "fs.s3a.jwt.path"; Review comment: If its for OIDC / WebIdentity, can we change to something like `fs.s3a.oidc.jwt.path` / `fs.s3a.webidentity.jwt.path`? + add `@value` to JavaDoc for IDE hints? ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java ## @@ -142,6 +142,10 @@ private Constants() { public static final String ASSUMED_ROLE_CREDENTIALS_DEFAULT = SimpleAWSCredentialsProvider.NAME; + /** + * Absolute path to the web identity token file + */ + public static final String JWT_PATH = "fs.s3a.jwt.path"; Review comment: If its for OIDC / WebIdentity, can we change to something like `fs.s3a.oidc.jwt.path` / `fs.s3a.webidentity.jwt.path`? Also, add `@value` to JavaDoc for IDE hints? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18154) S3A Authentication to support WebIdentity
[ https://issues.apache.org/jira/browse/HADOOP-18154?focusedWorklogId=745763&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745763 ] ASF GitHub Bot logged work on HADOOP-18154: --- Author: ASF GitHub Bot Created on: 22/Mar/22 12:00 Start Date: 22/Mar/22 12:00 Worklog Time Spent: 10m Work Description: dannycjones commented on a change in pull request #4070: URL: https://github.com/apache/hadoop/pull/4070#discussion_r832005639 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java ## @@ -142,6 +142,10 @@ private Constants() { public static final String ASSUMED_ROLE_CREDENTIALS_DEFAULT = SimpleAWSCredentialsProvider.NAME; + /** + * Absolute path to the web identity token file + */ + public static final String JWT_PATH = "fs.s3a.jwt.path"; Review comment: If its for OIDC / WebIdentity, can we change to something like `fs.s3a.oidc.jwt.path` / `fs.s3a.webidentity.jwt.path`? + add `@value` to JavaDoc for IDE hints? ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java ## @@ -142,6 +142,10 @@ private Constants() { public static final String ASSUMED_ROLE_CREDENTIALS_DEFAULT = SimpleAWSCredentialsProvider.NAME; + /** + * Absolute path to the web identity token file + */ + public static final String JWT_PATH = "fs.s3a.jwt.path"; Review comment: If its for OIDC / WebIdentity, can we change to something like `fs.s3a.oidc.jwt.path` / `fs.s3a.webidentity.jwt.path`? Also, add `@value` to JavaDoc for IDE hints? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745763) Time Spent: 2.5h (was: 2h 20m) > S3A Authentication to support WebIdentity > - > > Key: HADOOP-18154 > URL: https://issues.apache.org/jira/browse/HADOOP-18154 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 2.10.1 >Reporter: Ju Clarysse >Assignee: Ju Clarysse >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > We are using the latest version of > [delta-sharing|https://github.com/delta-io/delta-sharing] which takes > advantage of > [hadoop-aws|https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html] > (S3A) connector in [Hadoop release version > 2.10.1|https://github.com/apache/hadoop/tree/rel/release-2.10.1] to mount an > AWS S3 File System. In our particular setup, all services are operated in > Amazon Elastic Kubernetes Service (EKS) and need to comply to the AWS > security concept [IAM roles for service > accounts|https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html] > (IRSA). > As [Delta sharing S3 connection|https://github.com/delta-io/delta-sharing#s3] > doesn't offer any corresponding support, we patched hadoop-aws-2.10.1 to > address this need via a new credentials provider class > org.apache.hadoop.fs.s3a.OIDCTokenCredentialsProvider. We also upgraded > dependency aws-java-sdk-bundle to its latest version 1.12.167 as [AWS > WebIdentityTokenCredentialsProvider > class|https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/WebIdentityTokenCredentialsProvider.html%E2%80%A6] > was not yet available in original version 1.11.271. > We believe that other delta-sharing users could benefit from this short-term > contribution. Then sooner or later, delta-sharing owners will have to upgrade > their project to a more recent version of hadoop-aws that is probably more > widely used. The effort to promote this change is probably low. > Additional note: AWS WebIdentityTokenCredentialsProvider class is directly > supported by Spark applications submitted with configuration properties > `spark.hadoop.fs.s3a.aws.credentials.provider`and > `spark.kubernetes.authenticate.submission.oauthToken` > ([doc|https://spark.apache.org/docs/latest/running-on-kubernetes.html#spark-properties]). > So bringing this support to Hadoop will primarily be interesting for > non-Spark users. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] kiran-maturi commented on a change in pull request #3445: HADOOP-15566 Opentelemetry changes using java agent
kiran-maturi commented on a change in pull request #3445: URL: https://github.com/apache/hadoop/pull/3445#discussion_r832102387 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tracing/Span.java ## @@ -17,28 +17,54 @@ */ package org.apache.hadoop.tracing; +import io.opentelemetry.context.Scope; + import java.io.Closeable; public class Span implements Closeable { - + private io.opentelemetry.api.trace.Span span = null; public Span() { } + public Span(io.opentelemetry.api.trace.Span span){ +this.span = span; + } + public Span addKVAnnotation(String key, String value) { +if(span != null){ + span.setAttribute(key, value); +} return this; } public Span addTimelineAnnotation(String msg) { +if(span != null){ + span.addEvent(msg); +} return this; } public SpanContext getContext() { +if(span != null){ + return new SpanContext(span.getSpanContext()); +} return null; } public void finish() { +close(); } public void close() { +if(span != null){ + span.end(); Review comment: > would span need to be nullified here. or is it ok to invoke it after being ended? I think we need not nullify span after it has ended. In general we call span.end in a finally block so no further operation takes place. If we end the span before that and try adding events or call end it won't have any impact. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-15566) Support OpenTelemetry
[ https://issues.apache.org/jira/browse/HADOOP-15566?focusedWorklogId=745775&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745775 ] ASF GitHub Bot logged work on HADOOP-15566: --- Author: ASF GitHub Bot Created on: 22/Mar/22 12:15 Start Date: 22/Mar/22 12:15 Worklog Time Spent: 10m Work Description: kiran-maturi commented on a change in pull request #3445: URL: https://github.com/apache/hadoop/pull/3445#discussion_r832102387 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tracing/Span.java ## @@ -17,28 +17,54 @@ */ package org.apache.hadoop.tracing; +import io.opentelemetry.context.Scope; + import java.io.Closeable; public class Span implements Closeable { - + private io.opentelemetry.api.trace.Span span = null; public Span() { } + public Span(io.opentelemetry.api.trace.Span span){ +this.span = span; + } + public Span addKVAnnotation(String key, String value) { +if(span != null){ + span.setAttribute(key, value); +} return this; } public Span addTimelineAnnotation(String msg) { +if(span != null){ + span.addEvent(msg); +} return this; } public SpanContext getContext() { +if(span != null){ + return new SpanContext(span.getSpanContext()); +} return null; } public void finish() { +close(); } public void close() { +if(span != null){ + span.end(); Review comment: > would span need to be nullified here. or is it ok to invoke it after being ended? I think we need not nullify span after it has ended. In general we call span.end in a finally block so no further operation takes place. If we end the span before that and try adding events or call end it won't have any impact. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745775) Time Spent: 10h 20m (was: 10h 10m) > Support OpenTelemetry > - > > Key: HADOOP-15566 > URL: https://issues.apache.org/jira/browse/HADOOP-15566 > Project: Hadoop Common > Issue Type: New Feature > Components: metrics, tracing >Affects Versions: 3.1.0 >Reporter: Todd Lipcon >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available, security > Attachments: HADOOP-15566-WIP.1.patch, HADOOP-15566.000.WIP.patch, > OpenTelemetry Support Scope Doc v2.pdf, OpenTracing Support Scope Doc.pdf, > Screen Shot 2018-06-29 at 11.59.16 AM.png, ss-trace-s3a.png > > Time Spent: 10h 20m > Remaining Estimate: 0h > > The HTrace incubator project has voted to retire itself and won't be making > further releases. The Hadoop project currently has various hooks with HTrace. > It seems in some cases (eg HDFS-13702) these hooks have had measurable > performance overhead. Given these two factors, I think we should consider > removing the HTrace integration. If there is someone willing to do the work, > replacing it with OpenTracing might be a better choice since there is an > active community. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] 9uapaw closed pull request #4066: YARN-11087. Introduce the config to control the refresh interval in R…
9uapaw closed pull request #4066: URL: https://github.com/apache/hadoop/pull/4066 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] 9uapaw commented on pull request #4066: YARN-11087. Introduce the config to control the refresh interval in R…
9uapaw commented on pull request #4066: URL: https://github.com/apache/hadoop/pull/4066#issuecomment-1075102697 Thanks for the PR @zuston. Committed to trunk. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2584: HADOOP-16202. Enhance openFile() for better read performance against object stores
hadoop-yetus removed a comment on pull request #2584: URL: https://github.com/apache/hadoop/pull/2584#issuecomment-1025800148 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3445: HADOOP-15566 Opentelemetry changes using java agent
hadoop-yetus commented on pull request #3445: URL: https://github.com/apache/hadoop/pull/3445#issuecomment-1075137411 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 24s | | https://github.com/apache/hadoop/pull/3445 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/3445 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3445/8/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally
[ https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=745799&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745799 ] ASF GitHub Bot logged work on HADOOP-16202: --- Author: ASF GitHub Bot Created on: 22/Mar/22 12:52 Start Date: 22/Mar/22 12:52 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #2584: URL: https://github.com/apache/hadoop/pull/2584#issuecomment-1025800148 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745799) Time Spent: 14h 50m (was: 14h 40m) > Stabilize openFile() and adopt internally > - > > Key: HADOOP-16202 > URL: https://issues.apache.org/jira/browse/HADOOP-16202 > Project: Hadoop Common > Issue Type: Bug > Components: fs, fs/s3, tools/distcp >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 14h 50m > Remaining Estimate: 0h > > The {{openFile()}} builder API lets us add new options when reading a file > Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows > the length of the file to be declared. If set, *no check for the existence of > the file is issued when opening the file* > Also: withFileStatus() to take any FileStatus implementation, rather than > only S3AFileStatus -and not check that the path matches the path being > opened. Needed to support viewFS-style wrapping and mounting. > and Adopt where appropriate to stop clusters with S3A reads switched to > random IO from killing download/localization > * fs shell copyToLocal > * distcp > * IOUtils.copy -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-15566) Support OpenTelemetry
[ https://issues.apache.org/jira/browse/HADOOP-15566?focusedWorklogId=745801&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745801 ] ASF GitHub Bot logged work on HADOOP-15566: --- Author: ASF GitHub Bot Created on: 22/Mar/22 12:52 Start Date: 22/Mar/22 12:52 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #3445: URL: https://github.com/apache/hadoop/pull/3445#issuecomment-1075137411 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 24s | | https://github.com/apache/hadoop/pull/3445 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/3445 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3445/8/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745801) Time Spent: 10.5h (was: 10h 20m) > Support OpenTelemetry > - > > Key: HADOOP-15566 > URL: https://issues.apache.org/jira/browse/HADOOP-15566 > Project: Hadoop Common > Issue Type: New Feature > Components: metrics, tracing >Affects Versions: 3.1.0 >Reporter: Todd Lipcon >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available, security > Attachments: HADOOP-15566-WIP.1.patch, HADOOP-15566.000.WIP.patch, > OpenTelemetry Support Scope Doc v2.pdf, OpenTracing Support Scope Doc.pdf, > Screen Shot 2018-06-29 at 11.59.16 AM.png, ss-trace-s3a.png > > Time Spent: 10.5h > Remaining Estimate: 0h > > The HTrace incubator project has voted to retire itself and won't be making > further releases. The Hadoop project currently has various hooks with HTrace. > It seems in some cases (eg HDFS-13702) these hooks have had measurable > performance overhead. Given these two factors, I think we should consider > removing the HTrace integration. If there is someone willing to do the work, > replacing it with OpenTracing might be a better choice since there is an > active community. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2584: HADOOP-16202. Enhance openFile() for better read performance against object stores
steveloughran commented on pull request #2584: URL: https://github.com/apache/hadoop/pull/2584#issuecomment-1075138032 checkstyle ``` ./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:120:import org.apache.hadoop.fs.s3a.select.InternalSelectConstants;:8: Unused import - org.apache.hadoop.fs.s3a.select.InternalSelectConstants. [UnusedImports] ./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:194:import static org.apache.hadoop.fs.impl.AbstractFSBuilderImpl.rejectUnknownMandatoryKeys;:15: Unused import - org.apache.hadoop.fs.impl.AbstractFSBuilderImpl.rejectUnknownMandatoryKeys. [UnusedImports]' ``` javac warnings ``` hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:4099:14:[deprecation] getDefaultBlockSize() in FileSystem has been deprecated hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CopyFromLocalOperation.java:235:16:[unchecked] unchecked method invocation: method sort in interface List is applied to given types ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally
[ https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=745805&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745805 ] ASF GitHub Bot logged work on HADOOP-16202: --- Author: ASF GitHub Bot Created on: 22/Mar/22 12:53 Start Date: 22/Mar/22 12:53 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2584: URL: https://github.com/apache/hadoop/pull/2584#issuecomment-1075138032 checkstyle ``` ./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:120:import org.apache.hadoop.fs.s3a.select.InternalSelectConstants;:8: Unused import - org.apache.hadoop.fs.s3a.select.InternalSelectConstants. [UnusedImports] ./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:194:import static org.apache.hadoop.fs.impl.AbstractFSBuilderImpl.rejectUnknownMandatoryKeys;:15: Unused import - org.apache.hadoop.fs.impl.AbstractFSBuilderImpl.rejectUnknownMandatoryKeys. [UnusedImports]' ``` javac warnings ``` hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:4099:14:[deprecation] getDefaultBlockSize() in FileSystem has been deprecated hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CopyFromLocalOperation.java:235:16:[unchecked] unchecked method invocation: method sort in interface List is applied to given types ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745805) Time Spent: 15h (was: 14h 50m) > Stabilize openFile() and adopt internally > - > > Key: HADOOP-16202 > URL: https://issues.apache.org/jira/browse/HADOOP-16202 > Project: Hadoop Common > Issue Type: Bug > Components: fs, fs/s3, tools/distcp >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 15h > Remaining Estimate: 0h > > The {{openFile()}} builder API lets us add new options when reading a file > Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows > the length of the file to be declared. If set, *no check for the existence of > the file is issued when opening the file* > Also: withFileStatus() to take any FileStatus implementation, rather than > only S3AFileStatus -and not check that the path matches the path being > opened. Needed to support viewFS-style wrapping and mounting. > and Adopt where appropriate to stop clusters with S3A reads switched to > random IO from killing download/localization > * fs shell copyToLocal > * distcp > * IOUtils.copy -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3978: HADOOP-13704. Optimised getContentSummary()
hadoop-yetus removed a comment on pull request #3978: URL: https://github.com/apache/hadoop/pull/3978#issuecomment-1044978246 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13704) S3A getContentSummary() to move to listFiles(recursive) to count children; instrument use
[ https://issues.apache.org/jira/browse/HADOOP-13704?focusedWorklogId=745807&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745807 ] ASF GitHub Bot logged work on HADOOP-13704: --- Author: ASF GitHub Bot Created on: 22/Mar/22 12:55 Start Date: 22/Mar/22 12:55 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #3978: URL: https://github.com/apache/hadoop/pull/3978#issuecomment-1044978246 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745807) Time Spent: 2.5h (was: 2h 20m) > S3A getContentSummary() to move to listFiles(recursive) to count children; > instrument use > - > > Key: HADOOP-13704 > URL: https://issues.apache.org/jira/browse/HADOOP-13704 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Minor > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > Hive and a bit of Spark use {{getContentSummary()}} to get some summary stats > of a filesystem. This is very expensive on S3A (and any other object store), > especially as the base implementation does the recursive tree walk. > Because of HADOOP-13208, we have a full enumeration of files under a path > without directory costs...S3A can/should switch to this to speed up those > places where the operation is called. > Also > * API call needs FS spec and contract tests > * S3A could instrument invocation, so as to enable real-world popularity to > be measured -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] singer-bin commented on pull request #4069: HDFS-16457.Make fs.getspaceused.classname reconfigurable
singer-bin commented on pull request #4069: URL: https://github.com/apache/hadoop/pull/4069#issuecomment-1075184346 The failed check has nothing to do with my changes, the last check passed. @jojochuang ,As can be seen from the generated log, my ut passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] singer-bin commented on pull request #4069: HDFS-16457.Make fs.getspaceused.classname reconfigurable
singer-bin commented on pull request #4069: URL: https://github.com/apache/hadoop/pull/4069#issuecomment-1075198028 Can you review my code, thanks a lot. @jojochuang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing -bucket purged?
Steve Loughran created HADOOP-18168: --- Summary: ITestMarkerTool.testRunLimitedLandsatAudit failing -bucket purged? Key: HADOOP-18168 URL: https://issues.apache.org/jira/browse/HADOOP-18168 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3, test Affects Versions: 3.3.3 Reporter: Steve Loughran {{ITestMarkerTool.testRunLimitedLandsatAudit}} is failing -a scan which was meant to stop after the first page of results is finishing because there aren't so many objects there. first visible sign of the landsat-pds cleanup now we have requester pays, we could do this against another store with stability promises, e.g common crawl. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing -bucket purged?
[ https://issues.apache.org/jira/browse/HADOOP-18168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510496#comment-17510496 ] Steve Loughran commented on HADOOP-18168: - {code} [INFO] Running org.apache.hadoop.fs.s3a.ITestS3AConfiguration [ERROR] Tests run: 18, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 50.261 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.tools.ITestMarkerTool [ERROR] testRunLimitedLandsatAudit(org.apache.hadoop.fs.s3a.tools.ITestMarkerTool) Time elapsed: 2.9 s <<< FAILURE! java.lang.AssertionError: Expected an exception of type class org.apache.hadoop.util.ExitUtil$ExitException at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:409) at org.apache.hadoop.fs.s3a.s3guard.S3GuardToolTestHelper.runS3GuardCommandToFailure(S3GuardToolTestHelper.java:163) at org.apache.hadoop.fs.s3a.tools.AbstractMarkerToolTest.runToFailure(AbstractMarkerToolTest.java:271) at org.apache.hadoop.fs.s3a.tools.ITestMarkerTool.testRunLimitedLandsatAudit(ITestMarkerTool.java:320) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) {code} > ITestMarkerTool.testRunLimitedLandsatAudit failing -bucket purged? > -- > > Key: HADOOP-18168 > URL: https://issues.apache.org/jira/browse/HADOOP-18168 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.3 >Reporter: Steve Loughran >Priority: Minor > > {{ITestMarkerTool.testRunLimitedLandsatAudit}} is failing -a scan which was > meant to stop after the first page of results is finishing because there > aren't so many objects there. > > first visible sign of the landsat-pds cleanup > now we have requester pays, we could do this against another store with > stability promises, e.g common crawl. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3978: HADOOP-13704. Optimised getContentSummary()
steveloughran commented on pull request #3978: URL: https://github.com/apache/hadoop/pull/3978#issuecomment-1075201852 patch is in, though i realise i forgot to add the pr# to the header, which is automatic when done through the UI. never mind, at least the JIRA Is in. closing this PR as done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #3978: HADOOP-13704. Optimised getContentSummary()
steveloughran closed pull request #3978: URL: https://github.com/apache/hadoop/pull/3978 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3978: HADOOP-13704. Optimised getContentSummary()
hadoop-yetus removed a comment on pull request #3978: URL: https://github.com/apache/hadoop/pull/3978#issuecomment-1046907425 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 1m 11s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. | _ trunk Compile Tests _ | | +0 :ok: | mvndep | 12m 55s | | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 21m 43s | | trunk passed | | +1 :green_heart: | compile | 22m 28s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | compile | 19m 46s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | checkstyle | 3m 43s | | trunk passed | | +1 :green_heart: | mvnsite | 2m 40s | | trunk passed | | +1 :green_heart: | javadoc | 1m 57s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 2m 33s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 3m 48s | | trunk passed | | +1 :green_heart: | shadedclient | 21m 10s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 35s | | the patch passed | | +1 :green_heart: | compile | 21m 53s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javac | 21m 53s | | the patch passed | | +1 :green_heart: | compile | 19m 51s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | javac | 19m 51s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | +1 :green_heart: | checkstyle | 3m 32s | | the patch passed | | +1 :green_heart: | mvnsite | 2m 40s | | the patch passed | | +1 :green_heart: | javadoc | 1m 51s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 2m 34s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 4m 10s | | the patch passed | | +1 :green_heart: | shadedclient | 21m 22s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 32m 7s | | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 2m 46s | | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 59s | | The patch does not generate ASF License warnings. | | | | 233m 23s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/3978 | | Optional Tests | dupname asflicense mvnsite codespell markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle | | uname | Linux a16af9ec9b1c 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 945a9d9bfbe07a11e49cb325daaeb3bf05eb88ef | | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/5/testReport/ | | Max. process+thread count | 1664 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/5/console | | versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. -- This is an automated messag
[jira] [Work logged] (HADOOP-13704) S3A getContentSummary() to move to listFiles(recursive) to count children; instrument use
[ https://issues.apache.org/jira/browse/HADOOP-13704?focusedWorklogId=745832&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745832 ] ASF GitHub Bot logged work on HADOOP-13704: --- Author: ASF GitHub Bot Created on: 22/Mar/22 13:47 Start Date: 22/Mar/22 13:47 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #3978: URL: https://github.com/apache/hadoop/pull/3978#issuecomment-1075201852 patch is in, though i realise i forgot to add the pr# to the header, which is automatic when done through the UI. never mind, at least the JIRA Is in. closing this PR as done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745832) Time Spent: 2h 40m (was: 2.5h) > S3A getContentSummary() to move to listFiles(recursive) to count children; > instrument use > - > > Key: HADOOP-13704 > URL: https://issues.apache.org/jira/browse/HADOOP-13704 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Minor > Labels: pull-request-available > Time Spent: 2h 40m > Remaining Estimate: 0h > > Hive and a bit of Spark use {{getContentSummary()}} to get some summary stats > of a filesystem. This is very expensive on S3A (and any other object store), > especially as the base implementation does the recursive tree walk. > Because of HADOOP-13208, we have a full enumeration of files under a path > without directory costs...S3A can/should switch to this to speed up those > places where the operation is called. > Also > * API call needs FS spec and contract tests > * S3A could instrument invocation, so as to enable real-world popularity to > be measured -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13704) S3A getContentSummary() to move to listFiles(recursive) to count children; instrument use
[ https://issues.apache.org/jira/browse/HADOOP-13704?focusedWorklogId=745834&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745834 ] ASF GitHub Bot logged work on HADOOP-13704: --- Author: ASF GitHub Bot Created on: 22/Mar/22 13:47 Start Date: 22/Mar/22 13:47 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #3978: URL: https://github.com/apache/hadoop/pull/3978#issuecomment-1046907425 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 1m 11s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. | _ trunk Compile Tests _ | | +0 :ok: | mvndep | 12m 55s | | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 21m 43s | | trunk passed | | +1 :green_heart: | compile | 22m 28s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | compile | 19m 46s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | checkstyle | 3m 43s | | trunk passed | | +1 :green_heart: | mvnsite | 2m 40s | | trunk passed | | +1 :green_heart: | javadoc | 1m 57s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 2m 33s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 3m 48s | | trunk passed | | +1 :green_heart: | shadedclient | 21m 10s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 35s | | the patch passed | | +1 :green_heart: | compile | 21m 53s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javac | 21m 53s | | the patch passed | | +1 :green_heart: | compile | 19m 51s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | javac | 19m 51s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | +1 :green_heart: | checkstyle | 3m 32s | | the patch passed | | +1 :green_heart: | mvnsite | 2m 40s | | the patch passed | | +1 :green_heart: | javadoc | 1m 51s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 2m 34s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 4m 10s | | the patch passed | | +1 :green_heart: | shadedclient | 21m 22s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 32m 7s | | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 2m 46s | | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 59s | | The patch does not generate ASF License warnings. | | | | 233m 23s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/3978 | | Optional Tests | dupname asflicense mvnsite codespell markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle | | uname | Linux a16af9ec9b1c 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 945a9d9bfbe07a11e49cb325daaeb3bf05eb88ef | | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/5/testReport/ | | Max. process+thread count | 1664
[jira] [Work logged] (HADOOP-13704) S3A getContentSummary() to move to listFiles(recursive) to count children; instrument use
[ https://issues.apache.org/jira/browse/HADOOP-13704?focusedWorklogId=745833&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745833 ] ASF GitHub Bot logged work on HADOOP-13704: --- Author: ASF GitHub Bot Created on: 22/Mar/22 13:47 Start Date: 22/Mar/22 13:47 Worklog Time Spent: 10m Work Description: steveloughran closed pull request #3978: URL: https://github.com/apache/hadoop/pull/3978 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745833) Time Spent: 2h 50m (was: 2h 40m) > S3A getContentSummary() to move to listFiles(recursive) to count children; > instrument use > - > > Key: HADOOP-13704 > URL: https://issues.apache.org/jira/browse/HADOOP-13704 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Minor > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > Hive and a bit of Spark use {{getContentSummary()}} to get some summary stats > of a filesystem. This is very expensive on S3A (and any other object store), > especially as the base implementation does the recursive tree walk. > Because of HADOOP-13208, we have a full enumeration of files under a path > without directory costs...S3A can/should switch to this to speed up those > places where the operation is called. > Also > * API call needs FS spec and contract tests > * S3A could instrument invocation, so as to enable real-world popularity to > be measured -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13704) S3A getContentSummary() to move to listFiles(recursive) to count children; instrument use
[ https://issues.apache.org/jira/browse/HADOOP-13704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-13704: --- Assignee: Ahmar Suhail > S3A getContentSummary() to move to listFiles(recursive) to count children; > instrument use > - > > Key: HADOOP-13704 > URL: https://issues.apache.org/jira/browse/HADOOP-13704 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Ahmar Suhail >Priority: Minor > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > > Hive and a bit of Spark use {{getContentSummary()}} to get some summary stats > of a filesystem. This is very expensive on S3A (and any other object store), > especially as the base implementation does the recursive tree walk. > Because of HADOOP-13208, we have a full enumeration of files under a path > without directory costs...S3A can/should switch to this to speed up those > places where the operation is called. > Also > * API call needs FS spec and contract tests > * S3A could instrument invocation, so as to enable real-world popularity to > be measured -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13704) S3A getContentSummary() to move to listFiles(recursive) to count children; instrument use
[ https://issues.apache.org/jira/browse/HADOOP-13704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-13704. - Fix Version/s: 3.3.3 Resolution: Fixed fixed in 3.3.3. thanks! > S3A getContentSummary() to move to listFiles(recursive) to count children; > instrument use > - > > Key: HADOOP-13704 > URL: https://issues.apache.org/jira/browse/HADOOP-13704 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Ahmar Suhail >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.3 > > Time Spent: 3h > Remaining Estimate: 0h > > Hive and a bit of Spark use {{getContentSummary()}} to get some summary stats > of a filesystem. This is very expensive on S3A (and any other object store), > especially as the base implementation does the recursive tree walk. > Because of HADOOP-13208, we have a full enumeration of files under a path > without directory costs...S3A can/should switch to this to speed up those > places where the operation is called. > Also > * API call needs FS spec and contract tests > * S3A could instrument invocation, so as to enable real-world popularity to > be measured -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14837) Handle S3A "glacier" data
[ https://issues.apache.org/jira/browse/HADOOP-14837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510508#comment-17510508 ] Steve Loughran commented on HADOOP-14837: - good questions, -I have no idea what the right answers are bq. For reporting better, do we want to add in a new statistic, something like `objects_in_glacier` which will have the count of objects currently in glacier? why not? bq. In listings, we can add in a new option to filter out glacier files by doing something like `!summary.getStorageClass().equals("GLACIER")` in the acceptor here? After we do this and call `getContentSummary()` it won't return glacier files in the fileCount. I'm not worried about that. is the storage type returned in the list call. allowing it to be filtered there? i wouldn't want to do any HEAD requests here bq. getBlockLocations() there's special handling in spark for that location, which says "run your work anywnere". we doin't want to break that. I think the best tactic here is to work out what people I want to do here and provide the bare minimum. Looking at some of the JIRAs there's no consensus as to what people want. Do they want glaciated files to be skipped in queries? or for recovery to be triggered (somehow). Returning the storage type ARCHIVE would be enough for anyone who wants to identify these files (distcp?) and at least then know there's a cost in accessing them. > Handle S3A "glacier" data > - > > Key: HADOOP-14837 > URL: https://issues.apache.org/jira/browse/HADOOP-14837 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Priority: Minor > > SPARK-21797 covers how if you have AWS S3 set to copy some files to glacier, > they appear in the listing but GETs fail, and so does everything else > We should think about how best to handle this. > # report better > # if listings can identify files which are glaciated then maybe we could have > an option to filter them out > # test & see what happens -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing -bucket purged?
[ https://issues.apache.org/jira/browse/HADOOP-18168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510511#comment-17510511 ] Daniel Carl Jones commented on HADOOP-18168: Bucket definitely looks empty right now (2022-03-22 14:00). {noformat} ❯ aws s3 ls s3://landsat-pds/ 2016-08-19 18:12:04 105 robots.txt 2018-08-29 01:45:15 45603307 scene_list.gz{noformat} > ITestMarkerTool.testRunLimitedLandsatAudit failing -bucket purged? > -- > > Key: HADOOP-18168 > URL: https://issues.apache.org/jira/browse/HADOOP-18168 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.3 >Reporter: Steve Loughran >Priority: Minor > > {{ITestMarkerTool.testRunLimitedLandsatAudit}} is failing -a scan which was > meant to stop after the first page of results is finishing because there > aren't so many objects there. > > first visible sign of the landsat-pds cleanup > now we have requester pays, we could do this against another store with > stability promises, e.g common crawl. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #4036: HADOOP-18145.Decompress the ZIP file and retain the original file per…
steveloughran commented on a change in pull request #4036: URL: https://github.com/apache/hadoop/pull/4036#discussion_r832236126 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java ## @@ -667,6 +670,9 @@ public static void unZip(InputStream inputStream, File toDir) if (!file.setLastModified(entry.getTime())) { numOfFailedLastModifiedSet++; } + if (entry.getPlatform() == ZipArchiveEntry.PLATFORM_UNIX) { +Files.setPosixFilePermissions(file.toPath(), permissionsFromMode(entry.getUnixMode())); Review comment: ok -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18145) Fileutil's unzip method causes unzipped files to lose their original permissions
[ https://issues.apache.org/jira/browse/HADOOP-18145?focusedWorklogId=745847&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745847 ] ASF GitHub Bot logged work on HADOOP-18145: --- Author: ASF GitHub Bot Created on: 22/Mar/22 14:12 Start Date: 22/Mar/22 14:12 Worklog Time Spent: 10m Work Description: steveloughran commented on a change in pull request #4036: URL: https://github.com/apache/hadoop/pull/4036#discussion_r832236126 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java ## @@ -667,6 +670,9 @@ public static void unZip(InputStream inputStream, File toDir) if (!file.setLastModified(entry.getTime())) { numOfFailedLastModifiedSet++; } + if (entry.getPlatform() == ZipArchiveEntry.PLATFORM_UNIX) { +Files.setPosixFilePermissions(file.toPath(), permissionsFromMode(entry.getUnixMode())); Review comment: ok -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745847) Time Spent: 5.5h (was: 5h 20m) > Fileutil's unzip method causes unzipped files to lose their original > permissions > > > Key: HADOOP-18145 > URL: https://issues.apache.org/jira/browse/HADOOP-18145 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.7.2 >Reporter: jingxiong zhong >Priority: Major > Labels: pull-request-available > Time Spent: 5.5h > Remaining Estimate: 0h > > When Spark decompresses the zip file, if the original file has the executable > permission, but the unzip method of FileUtil is invoked, the decompressed > file loses the executable permission, we should save the original permission -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on a change in pull request #4085: HDFS-16511. Change some frequent method lock type in ReplicaMap.
Hexiaoqiao commented on a change in pull request #4085: URL: https://github.com/apache/hadoop/pull/4085#discussion_r832238320 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java ## @@ -602,6 +605,54 @@ public void run() {} + "volumeMap.", 0, totalNumReplicas); } + @Test(timeout = 3) + public void testCurrentWriteAndDeleteBlock() throws Exception { Review comment: testCurrentXXX -> testConcurrentXXX ? ## File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java ## @@ -602,6 +605,54 @@ public void run() {} + "volumeMap.", 0, totalNumReplicas); } + @Test(timeout = 3) + public void testCurrentWriteAndDeleteBlock() throws Exception { +// Feed FsDataset with block metadata. +final int numBlocks = 1000; +final int threadCount = 10; +// Generate data blocks. +ExecutorService pool = Executors.newFixedThreadPool(threadCount); +List> futureList = new ArrayList<>(); +for (int i = 0; i < threadCount; i++) { + Thread thread = new Thread() { +@Override +public void run() { + try { +for (int i = 0; i < numBlocks; i++) { + String bpid = BLOCK_POOL_IDS[numBlocks % BLOCK_POOL_IDS.length]; Review comment: The `numBlocks` and `BLOCK_POOL_IDS.length` are both static, so bpid is one certain value. Do you mean to random it? ## File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java ## @@ -602,6 +605,54 @@ public void run() {} + "volumeMap.", 0, totalNumReplicas); } + @Test(timeout = 3) + public void testCurrentWriteAndDeleteBlock() throws Exception { +// Feed FsDataset with block metadata. +final int numBlocks = 1000; +final int threadCount = 10; +// Generate data blocks. +ExecutorService pool = Executors.newFixedThreadPool(threadCount); +List> futureList = new ArrayList<>(); +for (int i = 0; i < threadCount; i++) { + Thread thread = new Thread() { +@Override +public void run() { + try { +for (int i = 0; i < numBlocks; i++) { + String bpid = BLOCK_POOL_IDS[numBlocks % BLOCK_POOL_IDS.length]; + ExtendedBlock eb = new ExtendedBlock(bpid, i); + ReplicaHandler replica = null; + try { +replica = dataset.createRbw(StorageType.DEFAULT, null, eb, +false); +if (i % 2 > 0) { + dataset.invalidate(bpid, new Block[]{eb.getLocalBlock()}); +} + } finally { +if (replica != null) { + replica.close(); +} + } +} + } catch (Exception e) { +e.printStackTrace(); + } +} + }; + thread.setName("AddBlock" + i); + futureList.add(pool.submit(thread)); +} +// Wait for data generation +for (Future f : futureList) { + f.get(); +} +int totalNumReplicas = 0; Review comment: Suggest to verify different blockpool's blocks num separately here. Sum them will loss some key information such as above comment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #2918: HDFS-15987. Improve oiv tool to parse fsimage file in parallel with d…
Hexiaoqiao commented on pull request #2918: URL: https://github.com/apache/hadoop/pull/2918#issuecomment-1075257224 Committed to trunk. Thanks @whbing for your contributions! Thanks all to reviews! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao closed pull request #2918: HDFS-15987. Improve oiv tool to parse fsimage file in parallel with d…
Hexiaoqiao closed pull request #2918: URL: https://github.com/apache/hadoop/pull/2918 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] rguillome closed pull request #426: correct configuration tag in mapred-site.xml
rguillome closed pull request #426: URL: https://github.com/apache/hadoop/pull/426 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] rguillome commented on pull request #426: correct configuration tag in mapred-site.xml
rguillome commented on pull request #426: URL: https://github.com/apache/hadoop/pull/426#issuecomment-1075277247 Corrected in 3.1.2 (https://issues.apache.org/jira/browse/MAPREDUCE-7165) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing -bucket purged?
[ https://issues.apache.org/jira/browse/HADOOP-18168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510511#comment-17510511 ] Daniel Carl Jones edited comment on HADOOP-18168 at 3/22/22, 4:17 PM: -- Current content of the landsat-pds bucket (2022-03-22 14:00). {noformat} ❯ aws s3 ls s3://landsat-pds/ 2016-08-19 18:12:04 105 robots.txt 2018-08-29 01:45:15 45603307 scene_list.gz{noformat} was (Author: JIRAUSER284792): Bucket definitely looks empty right now (2022-03-22 14:00). {noformat} ❯ aws s3 ls s3://landsat-pds/ 2016-08-19 18:12:04 105 robots.txt 2018-08-29 01:45:15 45603307 scene_list.gz{noformat} > ITestMarkerTool.testRunLimitedLandsatAudit failing -bucket purged? > -- > > Key: HADOOP-18168 > URL: https://issues.apache.org/jira/browse/HADOOP-18168 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.3 >Reporter: Steve Loughran >Priority: Minor > > {{ITestMarkerTool.testRunLimitedLandsatAudit}} is failing -a scan which was > meant to stop after the first page of results is finishing because there > aren't so many objects there. > > first visible sign of the landsat-pds cleanup > now we have requester pays, we could do this against another store with > stability promises, e.g common crawl. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] kiran-maturi commented on a change in pull request #3445: HADOOP-15566 Opentelemetry changes using java agent
kiran-maturi commented on a change in pull request #3445: URL: https://github.com/apache/hadoop/pull/3445#discussion_r832373680 ## File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/tracing/TestTracer.java ## @@ -0,0 +1,33 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.tracing; + +import org.junit.Test; + +import static org.junit.Assert.*; Review comment: @steveloughran sure will start adding more tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-15566) Support OpenTelemetry
[ https://issues.apache.org/jira/browse/HADOOP-15566?focusedWorklogId=745914&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745914 ] ASF GitHub Bot logged work on HADOOP-15566: --- Author: ASF GitHub Bot Created on: 22/Mar/22 16:26 Start Date: 22/Mar/22 16:26 Worklog Time Spent: 10m Work Description: kiran-maturi commented on a change in pull request #3445: URL: https://github.com/apache/hadoop/pull/3445#discussion_r832373680 ## File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/tracing/TestTracer.java ## @@ -0,0 +1,33 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.tracing; + +import org.junit.Test; + +import static org.junit.Assert.*; Review comment: @steveloughran sure will start adding more tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745914) Time Spent: 10h 40m (was: 10.5h) > Support OpenTelemetry > - > > Key: HADOOP-15566 > URL: https://issues.apache.org/jira/browse/HADOOP-15566 > Project: Hadoop Common > Issue Type: New Feature > Components: metrics, tracing >Affects Versions: 3.1.0 >Reporter: Todd Lipcon >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available, security > Attachments: HADOOP-15566-WIP.1.patch, HADOOP-15566.000.WIP.patch, > OpenTelemetry Support Scope Doc v2.pdf, OpenTracing Support Scope Doc.pdf, > Screen Shot 2018-06-29 at 11.59.16 AM.png, ss-trace-s3a.png > > Time Spent: 10h 40m > Remaining Estimate: 0h > > The HTrace incubator project has voted to retire itself and won't be making > further releases. The Hadoop project currently has various hooks with HTrace. > It seems in some cases (eg HDFS-13702) these hooks have had measurable > performance overhead. Given these two factors, I think we should consider > removing the HTrace integration. If there is someone willing to do the work, > replacing it with OpenTracing might be a better choice since there is an > active community. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing -bucket purged?
[ https://issues.apache.org/jira/browse/HADOOP-18168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510511#comment-17510511 ] Daniel Carl Jones edited comment on HADOOP-18168 at 3/22/22, 4:42 PM: -- Current content of the landsat-pds bucket (2022-03-22 14:00). {noformat} ❯ aws s3 ls s3://landsat-pds/ 2016-08-19 18:12:04 105 robots.txt 2018-08-29 01:45:15 45603307 scene_list.gz{noformat} Looks like _scene_list.gz_ is retained. However, ITestMarkerTool is relying on assumptions about the bucket's contents (large number of objects). was (Author: JIRAUSER284792): Current content of the landsat-pds bucket (2022-03-22 14:00). {noformat} ❯ aws s3 ls s3://landsat-pds/ 2016-08-19 18:12:04 105 robots.txt 2018-08-29 01:45:15 45603307 scene_list.gz{noformat} > ITestMarkerTool.testRunLimitedLandsatAudit failing -bucket purged? > -- > > Key: HADOOP-18168 > URL: https://issues.apache.org/jira/browse/HADOOP-18168 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.3 >Reporter: Steve Loughran >Priority: Minor > > {{ITestMarkerTool.testRunLimitedLandsatAudit}} is failing -a scan which was > meant to stop after the first page of results is finishing because there > aren't so many objects there. > > first visible sign of the landsat-pds cleanup > now we have requester pays, we could do this against another store with > stability promises, e.g common crawl. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #4036: HADOOP-18145.Decompress the ZIP file and retain the original file per…
hadoop-yetus removed a comment on pull request #4036: URL: https://github.com/apache/hadoop/pull/4036#issuecomment-1055349079 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 54s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 35m 12s | | trunk passed | | +1 :green_heart: | compile | 24m 17s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | compile | 20m 43s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | checkstyle | 0m 59s | | trunk passed | | +1 :green_heart: | mvnsite | 1m 38s | | trunk passed | | +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 1m 39s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 2m 30s | | trunk passed | | +1 :green_heart: | shadedclient | 25m 6s | | branch has no errors when building and testing our client artifacts. | | -0 :warning: | patch | 25m 27s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 59s | | the patch passed | | +1 :green_heart: | compile | 23m 38s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javac | 23m 38s | | the patch passed | | +1 :green_heart: | compile | 20m 47s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | javac | 20m 47s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | -0 :warning: | checkstyle | 1m 0s | [/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4036/8/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt) | hadoop-common-project/hadoop-common: The patch generated 3 new + 75 unchanged - 0 fixed = 78 total (was 75) | | +1 :green_heart: | mvnsite | 1m 35s | | the patch passed | | +1 :green_heart: | javadoc | 1m 3s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 1m 38s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 2m 42s | | the patch passed | | +1 :green_heart: | shadedclient | 25m 31s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 17m 43s | | hadoop-common in the patch passed. | | +1 :green_heart: | asflicense | 0m 49s | | The patch does not generate ASF License warnings. | | | | 211m 3s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4036/8/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4036 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell | | uname | Linux dc3cb049a2b0 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 7732357fbfae4404d331d08b9b790d4d752e4b86 | | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4036/8/testReport/ | | Max. process+thread count | 1252 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4036/8/console | | versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apac
[jira] [Work logged] (HADOOP-18145) Fileutil's unzip method causes unzipped files to lose their original permissions
[ https://issues.apache.org/jira/browse/HADOOP-18145?focusedWorklogId=745930&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745930 ] ASF GitHub Bot logged work on HADOOP-18145: --- Author: ASF GitHub Bot Created on: 22/Mar/22 16:43 Start Date: 22/Mar/22 16:43 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #4036: URL: https://github.com/apache/hadoop/pull/4036#issuecomment-1055349079 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 54s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 35m 12s | | trunk passed | | +1 :green_heart: | compile | 24m 17s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | compile | 20m 43s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | checkstyle | 0m 59s | | trunk passed | | +1 :green_heart: | mvnsite | 1m 38s | | trunk passed | | +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 1m 39s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 2m 30s | | trunk passed | | +1 :green_heart: | shadedclient | 25m 6s | | branch has no errors when building and testing our client artifacts. | | -0 :warning: | patch | 25m 27s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 59s | | the patch passed | | +1 :green_heart: | compile | 23m 38s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javac | 23m 38s | | the patch passed | | +1 :green_heart: | compile | 20m 47s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | javac | 20m 47s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | -0 :warning: | checkstyle | 1m 0s | [/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4036/8/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt) | hadoop-common-project/hadoop-common: The patch generated 3 new + 75 unchanged - 0 fixed = 78 total (was 75) | | +1 :green_heart: | mvnsite | 1m 35s | | the patch passed | | +1 :green_heart: | javadoc | 1m 3s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 1m 38s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 2m 42s | | the patch passed | | +1 :green_heart: | shadedclient | 25m 31s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 17m 43s | | hadoop-common in the patch passed. | | +1 :green_heart: | asflicense | 0m 49s | | The patch does not generate ASF License warnings. | | | | 211m 3s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4036/8/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4036 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell | | uname | Linux dc3cb049a2b0 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 7732357fbfae4404d331d08b9b790d4d752e4b86 | | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
[GitHub] [hadoop] steveloughran commented on a change in pull request #4036: HADOOP-18145.Decompress the ZIP file and retain the original file per…
steveloughran commented on a change in pull request #4036: URL: https://github.com/apache/hadoop/pull/4036#discussion_r832391092 ## File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java ## @@ -706,35 +706,40 @@ public void testCreateLocalTempFile() throws IOException { public void testUnZip() throws IOException { // make sa simple zip final File simpleZip = new File(del, FILE); -OutputStream os = new FileOutputStream(simpleZip); -ZipOutputStream tos = new ZipOutputStream(os); -try { - ZipEntry ze = new ZipEntry("foo"); - byte[] data = "some-content".getBytes("UTF-8"); - ze.setSize(data.length); - tos.putNextEntry(ze); - tos.write(data); - tos.closeEntry(); - tos.flush(); - tos.finish(); -} finally { - tos.close(); -} - -// successfully unzip it into an existing dir: -FileUtil.unZip(simpleZip, tmp); -// check result: -assertTrue(new File(tmp, "foo").exists()); -assertEquals(12, new File(tmp, "foo").length()); - -final File regularFile = new File(tmp, "QuickBrownFoxJumpsOverTheLazyDog"); -regularFile.createNewFile(); -assertTrue(regularFile.exists()); -try { - FileUtil.unZip(simpleZip, regularFile); - assertTrue("An IOException expected.", false); -} catch (IOException ioe) { - // okay +try (OutputStream os = new FileOutputStream(simpleZip); + ZipArchiveOutputStream tos = new ZipArchiveOutputStream(os)) { + try { +ZipArchiveEntry ze = new ZipArchiveEntry("foo"); +ze.setUnixMode(0555); +byte[] data = "some-content".getBytes("UTF-8"); +ze.setSize(data.length); +tos.putArchiveEntry(ze); +tos.write(data); +tos.closeArchiveEntry(); +tos.flush(); +tos.finish(); + } finally { +tos.close(); Review comment: this close will be automatic. if you make the reference on L710 ``` ZipArchiveOutputStream tos = new ZipArchiveOutputStream(new FileOutputStream(simpleZip)) ``` then everything will be lined up for automatic close ## File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java ## @@ -706,35 +706,40 @@ public void testCreateLocalTempFile() throws IOException { public void testUnZip() throws IOException { // make sa simple zip final File simpleZip = new File(del, FILE); -OutputStream os = new FileOutputStream(simpleZip); -ZipOutputStream tos = new ZipOutputStream(os); -try { - ZipEntry ze = new ZipEntry("foo"); - byte[] data = "some-content".getBytes("UTF-8"); - ze.setSize(data.length); - tos.putNextEntry(ze); - tos.write(data); - tos.closeEntry(); - tos.flush(); - tos.finish(); -} finally { - tos.close(); -} - -// successfully unzip it into an existing dir: -FileUtil.unZip(simpleZip, tmp); -// check result: -assertTrue(new File(tmp, "foo").exists()); -assertEquals(12, new File(tmp, "foo").length()); - -final File regularFile = new File(tmp, "QuickBrownFoxJumpsOverTheLazyDog"); -regularFile.createNewFile(); -assertTrue(regularFile.exists()); -try { - FileUtil.unZip(simpleZip, regularFile); - assertTrue("An IOException expected.", false); -} catch (IOException ioe) { - // okay +try (OutputStream os = new FileOutputStream(simpleZip); + ZipArchiveOutputStream tos = new ZipArchiveOutputStream(os)) { + try { +ZipArchiveEntry ze = new ZipArchiveEntry("foo"); +ze.setUnixMode(0555); +byte[] data = "some-content".getBytes("UTF-8"); +ze.setSize(data.length); +tos.putArchiveEntry(ze); +tos.write(data); +tos.closeArchiveEntry(); +tos.flush(); +tos.finish(); + } finally { +tos.close(); + } + + // successfully unzip it into an existing dir: + FileUtil.unZip(simpleZip, tmp); + // check result: + assertTrue(new File(tmp, "foo").exists()); + assertEquals(12, new File(tmp, "foo").length()); + assertTrue("file lacks execute permissions", new File(tmp, "foo").canExecute()); + assertFalse("file has write permissions", new File(tmp, "foo").canWrite()); + assertTrue("file lacks read permissions", new File(tmp, "foo").canRead()); + + final File regularFile = new File(tmp, "QuickBrownFoxJumpsOverTheLazyDog"); + regularFile.createNewFile(); + assertTrue(regularFile.exists()); + try { Review comment: use LambdaTestUtils.intercept() here. e.g. ```java LambdaTestUtils.intercept(IOException.class, () -> FileUtil.unZip(simpleZip, regularFile)); ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T
[jira] [Work logged] (HADOOP-18145) Fileutil's unzip method causes unzipped files to lose their original permissions
[ https://issues.apache.org/jira/browse/HADOOP-18145?focusedWorklogId=745931&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745931 ] ASF GitHub Bot logged work on HADOOP-18145: --- Author: ASF GitHub Bot Created on: 22/Mar/22 16:46 Start Date: 22/Mar/22 16:46 Worklog Time Spent: 10m Work Description: steveloughran commented on a change in pull request #4036: URL: https://github.com/apache/hadoop/pull/4036#discussion_r832391092 ## File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java ## @@ -706,35 +706,40 @@ public void testCreateLocalTempFile() throws IOException { public void testUnZip() throws IOException { // make sa simple zip final File simpleZip = new File(del, FILE); -OutputStream os = new FileOutputStream(simpleZip); -ZipOutputStream tos = new ZipOutputStream(os); -try { - ZipEntry ze = new ZipEntry("foo"); - byte[] data = "some-content".getBytes("UTF-8"); - ze.setSize(data.length); - tos.putNextEntry(ze); - tos.write(data); - tos.closeEntry(); - tos.flush(); - tos.finish(); -} finally { - tos.close(); -} - -// successfully unzip it into an existing dir: -FileUtil.unZip(simpleZip, tmp); -// check result: -assertTrue(new File(tmp, "foo").exists()); -assertEquals(12, new File(tmp, "foo").length()); - -final File regularFile = new File(tmp, "QuickBrownFoxJumpsOverTheLazyDog"); -regularFile.createNewFile(); -assertTrue(regularFile.exists()); -try { - FileUtil.unZip(simpleZip, regularFile); - assertTrue("An IOException expected.", false); -} catch (IOException ioe) { - // okay +try (OutputStream os = new FileOutputStream(simpleZip); + ZipArchiveOutputStream tos = new ZipArchiveOutputStream(os)) { + try { +ZipArchiveEntry ze = new ZipArchiveEntry("foo"); +ze.setUnixMode(0555); +byte[] data = "some-content".getBytes("UTF-8"); +ze.setSize(data.length); +tos.putArchiveEntry(ze); +tos.write(data); +tos.closeArchiveEntry(); +tos.flush(); +tos.finish(); + } finally { +tos.close(); Review comment: this close will be automatic. if you make the reference on L710 ``` ZipArchiveOutputStream tos = new ZipArchiveOutputStream(new FileOutputStream(simpleZip)) ``` then everything will be lined up for automatic close ## File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java ## @@ -706,35 +706,40 @@ public void testCreateLocalTempFile() throws IOException { public void testUnZip() throws IOException { // make sa simple zip final File simpleZip = new File(del, FILE); -OutputStream os = new FileOutputStream(simpleZip); -ZipOutputStream tos = new ZipOutputStream(os); -try { - ZipEntry ze = new ZipEntry("foo"); - byte[] data = "some-content".getBytes("UTF-8"); - ze.setSize(data.length); - tos.putNextEntry(ze); - tos.write(data); - tos.closeEntry(); - tos.flush(); - tos.finish(); -} finally { - tos.close(); -} - -// successfully unzip it into an existing dir: -FileUtil.unZip(simpleZip, tmp); -// check result: -assertTrue(new File(tmp, "foo").exists()); -assertEquals(12, new File(tmp, "foo").length()); - -final File regularFile = new File(tmp, "QuickBrownFoxJumpsOverTheLazyDog"); -regularFile.createNewFile(); -assertTrue(regularFile.exists()); -try { - FileUtil.unZip(simpleZip, regularFile); - assertTrue("An IOException expected.", false); -} catch (IOException ioe) { - // okay +try (OutputStream os = new FileOutputStream(simpleZip); + ZipArchiveOutputStream tos = new ZipArchiveOutputStream(os)) { + try { +ZipArchiveEntry ze = new ZipArchiveEntry("foo"); +ze.setUnixMode(0555); +byte[] data = "some-content".getBytes("UTF-8"); +ze.setSize(data.length); +tos.putArchiveEntry(ze); +tos.write(data); +tos.closeArchiveEntry(); +tos.flush(); +tos.finish(); + } finally { +tos.close(); + } + + // successfully unzip it into an existing dir: + FileUtil.unZip(simpleZip, tmp); + // check result: + assertTrue(new File(tmp, "foo").exists()); + assertEquals(12, new File(tmp, "foo").length()); + assertTrue("file lacks execute permissions", new File(tmp, "foo").canExecute()); + assertFalse("file has write permissions", new File(tmp, "foo").canWrite()); + assertTrue("file lacks read permissions", new File(tmp, "foo").canRead()); + + final File regularFile = new File(tmp, "QuickBrownFoxJumpsOverTheLazyDog"); + regularFile.create
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #4070: HADOOP-18154. S3A Authentication to support WebIdentity
hadoop-yetus removed a comment on pull request #4070: URL: https://github.com/apache/hadoop/pull/4070#issuecomment-1068327215 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18154) S3A Authentication to support WebIdentity
[ https://issues.apache.org/jira/browse/HADOOP-18154?focusedWorklogId=745945&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745945 ] ASF GitHub Bot logged work on HADOOP-18154: --- Author: ASF GitHub Bot Created on: 22/Mar/22 17:06 Start Date: 22/Mar/22 17:06 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #4070: URL: https://github.com/apache/hadoop/pull/4070#issuecomment-1068327215 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745945) Time Spent: 2h 40m (was: 2.5h) > S3A Authentication to support WebIdentity > - > > Key: HADOOP-18154 > URL: https://issues.apache.org/jira/browse/HADOOP-18154 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 2.10.1 >Reporter: Ju Clarysse >Assignee: Ju Clarysse >Priority: Major > Labels: pull-request-available > Time Spent: 2h 40m > Remaining Estimate: 0h > > We are using the latest version of > [delta-sharing|https://github.com/delta-io/delta-sharing] which takes > advantage of > [hadoop-aws|https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html] > (S3A) connector in [Hadoop release version > 2.10.1|https://github.com/apache/hadoop/tree/rel/release-2.10.1] to mount an > AWS S3 File System. In our particular setup, all services are operated in > Amazon Elastic Kubernetes Service (EKS) and need to comply to the AWS > security concept [IAM roles for service > accounts|https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html] > (IRSA). > As [Delta sharing S3 connection|https://github.com/delta-io/delta-sharing#s3] > doesn't offer any corresponding support, we patched hadoop-aws-2.10.1 to > address this need via a new credentials provider class > org.apache.hadoop.fs.s3a.OIDCTokenCredentialsProvider. We also upgraded > dependency aws-java-sdk-bundle to its latest version 1.12.167 as [AWS > WebIdentityTokenCredentialsProvider > class|https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/WebIdentityTokenCredentialsProvider.html%E2%80%A6] > was not yet available in original version 1.11.271. > We believe that other delta-sharing users could benefit from this short-term > contribution. Then sooner or later, delta-sharing owners will have to upgrade > their project to a more recent version of hadoop-aws that is probably more > widely used. The effort to promote this change is probably low. > Additional note: AWS WebIdentityTokenCredentialsProvider class is directly > supported by Spark applications submitted with configuration properties > `spark.hadoop.fs.s3a.aws.credentials.provider`and > `spark.kubernetes.authenticate.submission.oauthToken` > ([doc|https://spark.apache.org/docs/latest/running-on-kubernetes.html#spark-properties]). > So bringing this support to Hadoop will primarily be interesting for > non-Spark users. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #4074: HADOOP-18160 Avoid shading wildfly.openssl runtime dependency
hadoop-yetus removed a comment on pull request #4074: URL: https://github.com/apache/hadoop/pull/4074#issuecomment-1069468112 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 17m 34s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 37m 9s | | trunk passed | | +1 :green_heart: | compile | 0m 18s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | compile | 0m 18s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | mvnsite | 0m 22s | | trunk passed | | +1 :green_heart: | javadoc | 0m 19s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 0m 19s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | shadedclient | 61m 45s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 12s | | the patch passed | | +1 :green_heart: | compile | 0m 12s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javac | 0m 12s | | the patch passed | | +1 :green_heart: | compile | 0m 12s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | javac | 0m 12s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | +1 :green_heart: | mvnsite | 0m 14s | | the patch passed | | +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 0m 12s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 0m 12s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | shadedclient | 23m 36s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 0m 15s | | hadoop-client-api in the patch passed. | | +1 :green_heart: | asflicense | 0m 29s | | The patch does not generate ASF License warnings. | | | | 109m 5s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4074/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4074 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml | | uname | Linux 629688fc11e8 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 427a93ab844fa863e25f1893b105af9ee2c3cbfb | | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4074/2/testReport/ | | Max. process+thread count | 519 (vs. ulimit of 5500) | | modules | C: hadoop-client-modules/hadoop-client-api U: hadoop-client-modules/hadoop-client-api | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4074/2/console | | versions | git=2.25.1 maven=3.6.3 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@had
[jira] [Work logged] (HADOOP-18160) `org.wildfly.openssl` should not be shaded by Hadoop build
[ https://issues.apache.org/jira/browse/HADOOP-18160?focusedWorklogId=745949&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745949 ] ASF GitHub Bot logged work on HADOOP-18160: --- Author: ASF GitHub Bot Created on: 22/Mar/22 17:12 Start Date: 22/Mar/22 17:12 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #4074: URL: https://github.com/apache/hadoop/pull/4074#issuecomment-1069468112 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 17m 34s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 37m 9s | | trunk passed | | +1 :green_heart: | compile | 0m 18s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | compile | 0m 18s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | mvnsite | 0m 22s | | trunk passed | | +1 :green_heart: | javadoc | 0m 19s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 0m 19s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | shadedclient | 61m 45s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 12s | | the patch passed | | +1 :green_heart: | compile | 0m 12s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javac | 0m 12s | | the patch passed | | +1 :green_heart: | compile | 0m 12s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | javac | 0m 12s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | +1 :green_heart: | mvnsite | 0m 14s | | the patch passed | | +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 0m 12s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 0m 12s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | shadedclient | 23m 36s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 0m 15s | | hadoop-client-api in the patch passed. | | +1 :green_heart: | asflicense | 0m 29s | | The patch does not generate ASF License warnings. | | | | 109m 5s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4074/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4074 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml | | uname | Linux 629688fc11e8 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 427a93ab844fa863e25f1893b105af9ee2c3cbfb | | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4074/2/testReport/ | | Max. process+thread count | 519 (vs. ulimit of 5500) | | modules | C: hadoop-client-modules/hadoop-client-api U: hadoop-client-modules/hadoop-client-api | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4074/2/console | | versions | git=2.25.1 maven=3.6.3 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. -- This is an aut
[GitHub] [hadoop] omalley opened a new pull request #4091: HDFS-16517 Distance metric is wrong for non-DN machines in 2.10.
omalley opened a new pull request #4091: URL: https://github.com/apache/hadoop/pull/4091 ### Description of PR The distance metric used for machines in 2.10 that aren't in the NetworkTopology, because they aren't running DataNodes, is wrong. It means that off-rack and on-rack, but off-node, are both given a weight of 2. In normal Hadoop clusters, this isn't a big problem because they don't have clients that are on-rack but without DataNodes. For clusters that are striped (federated HDFS going across racks) or separate compute and storage that share racks are both really bad with this bug. ### How was this patch tested? Unit test added. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #4070: HADOOP-18154. S3A Authentication to support WebIdentity
steveloughran commented on pull request #4070: URL: https://github.com/apache/hadoop/pull/4070#issuecomment-1075401037 Looking at the `WebIdentityTokenCredentialsProvider` I see that if it doesn't get the parameters then it will fall back to environment variables. We absolutely do not want to be picking up env vars as it will only create support issues where configurations only work on a certain machines. (actually, we can ignore the session name settings as they are harmless) I'm going to propose we go with @dannycjones's suggestion and support the whole set of values and have the prefix `fs.s3a.webidentity` for all of them. for the arn, we could have a property `fs.s3a.webidentity.role.arn` but, what should we do if it wasn't set? 1. fail to initialize 2. have that null value force the env var lookup. I don't see any way to a completely block the environment variable resolution, which is a pain. I also see in the internal Library classes that sometimes roles are set up with an external ID, but it is not possible here. Is that an issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18154) S3A Authentication to support WebIdentity
[ https://issues.apache.org/jira/browse/HADOOP-18154?focusedWorklogId=745954&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745954 ] ASF GitHub Bot logged work on HADOOP-18154: --- Author: ASF GitHub Bot Created on: 22/Mar/22 17:22 Start Date: 22/Mar/22 17:22 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #4070: URL: https://github.com/apache/hadoop/pull/4070#issuecomment-1075401037 Looking at the `WebIdentityTokenCredentialsProvider` I see that if it doesn't get the parameters then it will fall back to environment variables. We absolutely do not want to be picking up env vars as it will only create support issues where configurations only work on a certain machines. (actually, we can ignore the session name settings as they are harmless) I'm going to propose we go with @dannycjones's suggestion and support the whole set of values and have the prefix `fs.s3a.webidentity` for all of them. for the arn, we could have a property `fs.s3a.webidentity.role.arn` but, what should we do if it wasn't set? 1. fail to initialize 2. have that null value force the env var lookup. I don't see any way to a completely block the environment variable resolution, which is a pain. I also see in the internal Library classes that sometimes roles are set up with an external ID, but it is not possible here. Is that an issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745954) Time Spent: 2h 50m (was: 2h 40m) > S3A Authentication to support WebIdentity > - > > Key: HADOOP-18154 > URL: https://issues.apache.org/jira/browse/HADOOP-18154 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 2.10.1 >Reporter: Ju Clarysse >Assignee: Ju Clarysse >Priority: Major > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > We are using the latest version of > [delta-sharing|https://github.com/delta-io/delta-sharing] which takes > advantage of > [hadoop-aws|https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html] > (S3A) connector in [Hadoop release version > 2.10.1|https://github.com/apache/hadoop/tree/rel/release-2.10.1] to mount an > AWS S3 File System. In our particular setup, all services are operated in > Amazon Elastic Kubernetes Service (EKS) and need to comply to the AWS > security concept [IAM roles for service > accounts|https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html] > (IRSA). > As [Delta sharing S3 connection|https://github.com/delta-io/delta-sharing#s3] > doesn't offer any corresponding support, we patched hadoop-aws-2.10.1 to > address this need via a new credentials provider class > org.apache.hadoop.fs.s3a.OIDCTokenCredentialsProvider. We also upgraded > dependency aws-java-sdk-bundle to its latest version 1.12.167 as [AWS > WebIdentityTokenCredentialsProvider > class|https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/WebIdentityTokenCredentialsProvider.html%E2%80%A6] > was not yet available in original version 1.11.271. > We believe that other delta-sharing users could benefit from this short-term > contribution. Then sooner or later, delta-sharing owners will have to upgrade > their project to a more recent version of hadoop-aws that is probably more > widely used. The effort to promote this change is probably low. > Additional note: AWS WebIdentityTokenCredentialsProvider class is directly > supported by Spark applications submitted with configuration properties > `spark.hadoop.fs.s3a.aws.credentials.provider`and > `spark.kubernetes.authenticate.submission.oauthToken` > ([doc|https://spark.apache.org/docs/latest/running-on-kubernetes.html#spark-properties]). > So bringing this support to Hadoop will primarily be interesting for > non-Spark users. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18167) Add metrics to track delegation token secret manager operations
[ https://issues.apache.org/jira/browse/HADOOP-18167?focusedWorklogId=745957&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745957 ] ASF GitHub Bot logged work on HADOOP-18167: --- Author: ASF GitHub Bot Created on: 22/Mar/22 17:27 Start Date: 22/Mar/22 17:27 Worklog Time Spent: 10m Work Description: hchaverri opened a new pull request #4092: URL: https://github.com/apache/hadoop/pull/4092 …erations ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745957) Remaining Estimate: 0h Time Spent: 10m > Add metrics to track delegation token secret manager operations > --- > > Key: HADOOP-18167 > URL: https://issues.apache.org/jira/browse/HADOOP-18167 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > New metrics to track operations that store, update and remove delegation > tokens in implementations of AbstractDelegationTokenSecretManager. This will > help evaluate the impact of using different secret managers and add > optimizations. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hchaverri opened a new pull request #4092: HADOOP-18167. Add metrics to track delegation token secret manager op…
hchaverri opened a new pull request #4092: URL: https://github.com/apache/hadoop/pull/4092 …erations ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18167) Add metrics to track delegation token secret manager operations
[ https://issues.apache.org/jira/browse/HADOOP-18167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18167: Labels: pull-request-available (was: ) > Add metrics to track delegation token secret manager operations > --- > > Key: HADOOP-18167 > URL: https://issues.apache.org/jira/browse/HADOOP-18167 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > New metrics to track operations that store, update and remove delegation > tokens in implementations of AbstractDelegationTokenSecretManager. This will > help evaluate the impact of using different secret managers and add > optimizations. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #4074: HADOOP-18160 Avoid shading wildfly.openssl runtime dependency
steveloughran merged pull request #4074: URL: https://github.com/apache/hadoop/pull/4074 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18160) `org.wildfly.openssl` should not be shaded by Hadoop build
[ https://issues.apache.org/jira/browse/HADOOP-18160?focusedWorklogId=745961&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745961 ] ASF GitHub Bot logged work on HADOOP-18160: --- Author: ASF GitHub Bot Created on: 22/Mar/22 17:32 Start Date: 22/Mar/22 17:32 Worklog Time Spent: 10m Work Description: steveloughran merged pull request #4074: URL: https://github.com/apache/hadoop/pull/4074 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745961) Time Spent: 1h 20m (was: 1h 10m) > `org.wildfly.openssl` should not be shaded by Hadoop build > -- > > Key: HADOOP-18160 > URL: https://issues.apache.org/jira/browse/HADOOP-18160 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.3.1 > Environment: hadoop 3.3.1 > spark 3.2.1 > JDK8 >Reporter: André F. >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > `org.wildfly.openssl` is a runtime library and its references are being > shaded on Hadoop, breaking the integration with other frameworks like Spark, > whenever the "fs.s3a.ssl.channel.mode" is set to "openssl". The error > produced in this situation is: > {code:java} > Suppressed: java.lang.NoClassDefFoundError: > org/apache/hadoop/shaded/org/wildfly/openssl/OpenSSLProvider{code} > Whenever it tries to be instantiated from the `DelegatingSSLSocketFactory`. > Spark tries to add it to its classpath without the shade, thus creating this > issue. > Dependencies which are not on "compile" scope should probably not be shaded > to avoid this kind of integration issues. > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] guptashailesh92 opened a new pull request #4093: YARN-11093: fix for reading files in timeline server in chronological order in case of fs-support-append to false
guptashailesh92 opened a new pull request #4093: URL: https://github.com/apache/hadoop/pull/4093 ### Description of PR In our setup, we are using Hive and Tez and using offline tez-ui mode, where we copy the ATS events files to other place and start tez-ui. To keep the event files small, we are using fs-support-append = false, and creating a new file for writing the events. We can see that, in this mode, file name contains [suffix](https://github.com/apache/hadoop/blob/1d5650c4d0acf33f141d593762682b3603523104/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/FileSystemTimelineWriter.java#L382) as timestamp. But at the time of [read](https://github.com/apache/hadoop/blob/1d5650c4d0acf33f141d593762682b3603523104/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java#L844), we are relying on the FS to provide correct file order. If that fails, we are adding event out of order, leading to event being discarded, or incorrect information. Fix could be sorting of the file names, based on suffix if append mode is not used. sample file names: summarylog-appattempt_1647348120288_0001_01_460237 entitylog-timelineEntityGroupId_1647348120288_1_dag_1647348120288_0001_1_673147 ### How was this patch tested? 1. Added conf `yarn.timeline-service.fs-support-append` to false. 2. Ran a job, which created small files for domain, entity and summary log. 3. Ran ATS by copying all the entity files to custom location, and checked tez-ui with and without the change. Was able to see latest status of the task correctly after the change. ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18169) getDelegationToken in ViewFs should also fetch the token from the fallback FS
Xing Lin created HADOOP-18169: - Summary: getDelegationToken in ViewFs should also fetch the token from the fallback FS Key: HADOOP-18169 URL: https://issues.apache.org/jira/browse/HADOOP-18169 Project: Hadoop Common Issue Type: Bug Reporter: Xing Lin getDelegationToken in ViewFs does not include the delegationToken from the fallback FS, while it should. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18169) getDelegationTokens in ViewFs should also fetch the token from the fallback FS
[ https://issues.apache.org/jira/browse/HADOOP-18169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xing Lin updated HADOOP-18169: -- Description: getDelegationTokens in ViewFs does not include the delegationToken from the fallback FS, while it should. (was: getDelegationToken in ViewFs does not include the delegationToken from the fallback FS, while it should. ) > getDelegationTokens in ViewFs should also fetch the token from the fallback FS > -- > > Key: HADOOP-18169 > URL: https://issues.apache.org/jira/browse/HADOOP-18169 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xing Lin >Priority: Major > > getDelegationTokens in ViewFs does not include the delegationToken from the > fallback FS, while it should. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18169) getDelegationTokens in ViewFs should also fetch the token from the fallback FS
[ https://issues.apache.org/jira/browse/HADOOP-18169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xing Lin updated HADOOP-18169: -- Summary: getDelegationTokens in ViewFs should also fetch the token from the fallback FS (was: getDelegationToken in ViewFs should also fetch the token from the fallback FS) > getDelegationTokens in ViewFs should also fetch the token from the fallback FS > -- > > Key: HADOOP-18169 > URL: https://issues.apache.org/jira/browse/HADOOP-18169 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xing Lin >Priority: Major > > getDelegationToken in ViewFs does not include the delegationToken from the > fallback FS, while it should. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] dannycjones commented on pull request #3962: HADOOP-14661. Add S3 requester pays bucket support to S3A
dannycjones commented on pull request #3962: URL: https://github.com/apache/hadoop/pull/3962#issuecomment-1075435015 @steveloughran I caught those after I pushed most of the other fixes from the last feedback and pushed a second commit ae8e812e11148bcf007d51fbf94356229df4fae1. There's two Yetus comments, latest one seems happy. Are we good to merge? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] GauthamBanasandra merged pull request #4086: HDFS-16471. Make HDFS ls tool cross platform
GauthamBanasandra merged pull request #4086: URL: https://github.com/apache/hadoop/pull/4086 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-14661) S3A to support Requester Pays Buckets
[ https://issues.apache.org/jira/browse/HADOOP-14661?focusedWorklogId=745990&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745990 ] ASF GitHub Bot logged work on HADOOP-14661: --- Author: ASF GitHub Bot Created on: 22/Mar/22 17:57 Start Date: 22/Mar/22 17:57 Worklog Time Spent: 10m Work Description: dannycjones commented on pull request #3962: URL: https://github.com/apache/hadoop/pull/3962#issuecomment-1075435015 @steveloughran I caught those after I pushed most of the other fixes from the last feedback and pushed a second commit ae8e812e11148bcf007d51fbf94356229df4fae1. There's two Yetus comments, latest one seems happy. Are we good to merge? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745990) Time Spent: 5h 50m (was: 5h 40m) > S3A to support Requester Pays Buckets > - > > Key: HADOOP-14661 > URL: https://issues.apache.org/jira/browse/HADOOP-14661 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, util >Affects Versions: 3.0.0-alpha3 >Reporter: Mandus Momberg >Assignee: Mandus Momberg >Priority: Minor > Labels: pull-request-available > Attachments: HADOOP-14661.patch > > Original Estimate: 2h > Time Spent: 5h 50m > Remaining Estimate: 0h > > Amazon S3 has the ability to charge the requester for the cost of accessing > S3. This is called Requester Pays Buckets. > In order to access these buckets, each request needs to be signed with a > specific header. > http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xinglin opened a new pull request #4094: HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS
xinglin opened a new pull request #4094: URL: https://github.com/apache/hadoop/pull/4094 ### Description of PR ### How was this patch tested? mvn test -Dtest="TestViewFs*" in both hadoop-common-project and hadoop-hdfs-project. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18169) getDelegationTokens in ViewFs should also fetch the token from the fallback FS
[ https://issues.apache.org/jira/browse/HADOOP-18169?focusedWorklogId=745998&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745998 ] ASF GitHub Bot logged work on HADOOP-18169: --- Author: ASF GitHub Bot Created on: 22/Mar/22 18:04 Start Date: 22/Mar/22 18:04 Worklog Time Spent: 10m Work Description: xinglin opened a new pull request #4094: URL: https://github.com/apache/hadoop/pull/4094 ### Description of PR ### How was this patch tested? mvn test -Dtest="TestViewFs*" in both hadoop-common-project and hadoop-hdfs-project. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745998) Remaining Estimate: 0h Time Spent: 10m > getDelegationTokens in ViewFs should also fetch the token from the fallback FS > -- > > Key: HADOOP-18169 > URL: https://issues.apache.org/jira/browse/HADOOP-18169 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xing Lin >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > getDelegationTokens in ViewFs does not include the delegationToken from the > fallback FS, while it should. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18169) getDelegationTokens in ViewFs should also fetch the token from the fallback FS
[ https://issues.apache.org/jira/browse/HADOOP-18169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18169: Labels: pull-request-available (was: ) > getDelegationTokens in ViewFs should also fetch the token from the fallback FS > -- > > Key: HADOOP-18169 > URL: https://issues.apache.org/jira/browse/HADOOP-18169 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xing Lin >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > getDelegationTokens in ViewFs does not include the delegationToken from the > fallback FS, while it should. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mccormickt12 commented on a change in pull request #4094: HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS
mccormickt12 commented on a change in pull request #4094: URL: https://github.com/apache/hadoop/pull/4094#discussion_r832492192 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java ## @@ -746,6 +746,17 @@ public void setVerifyChecksum(final boolean verifyChecksum) result.addAll(tokens); } } + +// Add tokens from fallback FS +if (this.fsState.getRootFallbackLink() != null) { + AbstractFileSystem linkedFallbackFs = Review comment: nit: these should be linkFallbackFs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4060: YARN-11084. Introduce new config to specify AM default node-label whe…
hadoop-yetus commented on pull request #4060: URL: https://github.com/apache/hadoop/pull/4060#issuecomment-1075471974 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 2m 25s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. | _ trunk Compile Tests _ | | +0 :ok: | mvndep | 12m 39s | | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 27m 8s | | trunk passed | | +1 :green_heart: | compile | 11m 8s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | compile | 9m 1s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | checkstyle | 2m 14s | | trunk passed | | +1 :green_heart: | mvnsite | 3m 58s | | trunk passed | | +1 :green_heart: | javadoc | 3m 4s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 2m 52s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +0 :ok: | spotbugs | 0m 27s | | branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site no spotbugs output file (spotbugsXml.xml) | | +1 :green_heart: | shadedclient | 24m 4s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 25s | | the patch passed | | +1 :green_heart: | compile | 10m 34s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javac | 10m 34s | | the patch passed | | +1 :green_heart: | compile | 9m 0s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | javac | 9m 0s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | -0 :warning: | checkstyle | 1m 37s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4060/7/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt) | hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 229 unchanged - 0 fixed = 230 total (was 229) | | +1 :green_heart: | mvnsite | 3m 18s | | the patch passed | | +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 2m 46s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 2m 39s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +0 :ok: | spotbugs | 0m 24s | | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site has no data from spotbugs | | +1 :green_heart: | shadedclient | 23m 8s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 1m 2s | | hadoop-yarn-api in the patch passed. | | +1 :green_heart: | unit | 4m 43s | | hadoop-yarn-common in the patch passed. | | +1 :green_heart: | unit | 100m 46s | | hadoop-yarn-server-resourcemanager in the patch passed. | | +1 :green_heart: | unit | 0m 24s | | hadoop-yarn-site in the patch passed. | | +1 :green_heart: | asflicense | 0m 46s | | The patch does not generate ASF License warnings. | | | | 280m 0s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4060/7/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4060 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml markdownlint | | uname | Linux f3f6bee8b187 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 147c2d492aa61854011dff48ef8017bd5c017cce | | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64
[jira] [Work logged] (HADOOP-18169) getDelegationTokens in ViewFs should also fetch the token from the fallback FS
[ https://issues.apache.org/jira/browse/HADOOP-18169?focusedWorklogId=746014&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746014 ] ASF GitHub Bot logged work on HADOOP-18169: --- Author: ASF GitHub Bot Created on: 22/Mar/22 18:25 Start Date: 22/Mar/22 18:25 Worklog Time Spent: 10m Work Description: mccormickt12 commented on a change in pull request #4094: URL: https://github.com/apache/hadoop/pull/4094#discussion_r832492192 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java ## @@ -746,6 +746,17 @@ public void setVerifyChecksum(final boolean verifyChecksum) result.addAll(tokens); } } + +// Add tokens from fallback FS +if (this.fsState.getRootFallbackLink() != null) { + AbstractFileSystem linkedFallbackFs = Review comment: nit: these should be linkFallbackFs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 746014) Time Spent: 20m (was: 10m) > getDelegationTokens in ViewFs should also fetch the token from the fallback FS > -- > > Key: HADOOP-18169 > URL: https://issues.apache.org/jira/browse/HADOOP-18169 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xing Lin >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > getDelegationTokens in ViewFs does not include the delegationToken from the > fallback FS, while it should. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18160) `org.wildfly.openssl` should not be shaded by Hadoop build
[ https://issues.apache.org/jira/browse/HADOOP-18160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18160. - Fix Version/s: 3.3.3 Resolution: Fixed > `org.wildfly.openssl` should not be shaded by Hadoop build > -- > > Key: HADOOP-18160 > URL: https://issues.apache.org/jira/browse/HADOOP-18160 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.3.1 > Environment: hadoop 3.3.1 > spark 3.2.1 > JDK8 >Reporter: André F. >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.3 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > `org.wildfly.openssl` is a runtime library and its references are being > shaded on Hadoop, breaking the integration with other frameworks like Spark, > whenever the "fs.s3a.ssl.channel.mode" is set to "openssl". The error > produced in this situation is: > {code:java} > Suppressed: java.lang.NoClassDefFoundError: > org/apache/hadoop/shaded/org/wildfly/openssl/OpenSSLProvider{code} > Whenever it tries to be instantiated from the `DelegatingSSLSocketFactory`. > Spark tries to add it to its classpath without the shade, thus creating this > issue. > Dependencies which are not on "compile" scope should probably not be shaded > to avoid this kind of integration issues. > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18160) `org.wildfly.openssl` should not be shaded by Hadoop build
[ https://issues.apache.org/jira/browse/HADOOP-18160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-18160: --- Assignee: André F. > `org.wildfly.openssl` should not be shaded by Hadoop build > -- > > Key: HADOOP-18160 > URL: https://issues.apache.org/jira/browse/HADOOP-18160 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.3.1 > Environment: hadoop 3.3.1 > spark 3.2.1 > JDK8 >Reporter: André F. >Assignee: André F. >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.3 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > `org.wildfly.openssl` is a runtime library and its references are being > shaded on Hadoop, breaking the integration with other frameworks like Spark, > whenever the "fs.s3a.ssl.channel.mode" is set to "openssl". The error > produced in this situation is: > {code:java} > Suppressed: java.lang.NoClassDefFoundError: > org/apache/hadoop/shaded/org/wildfly/openssl/OpenSSLProvider{code} > Whenever it tries to be instantiated from the `DelegatingSSLSocketFactory`. > Spark tries to add it to its classpath without the shade, thus creating this > issue. > Dependencies which are not on "compile" scope should probably not be shaded > to avoid this kind of integration issues. > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3470: YARN-10965. Centralize queue resource calculation based on CapacityVectors
hadoop-yetus commented on pull request #3470: URL: https://github.com/apache/hadoop/pull/3470#issuecomment-1075518999 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 37s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. | _ trunk Compile Tests _ | | +0 :ok: | mvndep | 12m 40s | | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 22m 58s | | trunk passed | | +1 :green_heart: | compile | 9m 45s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | compile | 8m 26s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | checkstyle | 1m 44s | | trunk passed | | +1 :green_heart: | mvnsite | 2m 18s | | trunk passed | | +1 :green_heart: | javadoc | 1m 59s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 1m 50s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 4m 13s | | trunk passed | | +1 :green_heart: | shadedclient | 22m 13s | | branch has no errors when building and testing our client artifacts. | | -0 :warning: | patch | 22m 35s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 31s | | the patch passed | | +1 :green_heart: | compile | 9m 2s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | -1 :x: | javac | 9m 2s | [/results-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3470/14/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt) | hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 392 unchanged - 1 fixed = 393 total (was 393) | | +1 :green_heart: | compile | 8m 18s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | javac | 8m 18s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | -0 :warning: | checkstyle | 1m 38s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3470/14/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt) | hadoop-yarn-project/hadoop-yarn: The patch generated 92 new + 155 unchanged - 0 fixed = 247 total (was 155) | | +1 :green_heart: | mvnsite | 2m 8s | | the patch passed | | +1 :green_heart: | javadoc | 1m 46s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 1m 41s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | -1 :x: | spotbugs | 2m 13s | [/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3470/14/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 :green_heart: | shadedclient | 21m 32s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 1m 7s | | hadoop-yarn-api in the patch passed. | | +1 :green_heart: | unit | 97m 12s | | hadoop-yarn-server-resourcemanager in the patch passed. | | +1 :green_heart: | asflicense | 0m 51s | | The patch does not generate ASF License warnings. | | | | 241m 30s | | | | Reason | Tests | |---:|:--| | SpotBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager | | | org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ResourceCalculationDriver.CALCULATO
[GitHub] [hadoop] steveloughran commented on pull request #4074: HADOOP-18160 Avoid shading wildfly.openssl runtime dependency
steveloughran commented on pull request #4074: URL: https://github.com/apache/hadoop/pull/4074#issuecomment-1075521131 ok, merged in to branch-3.3 and trunk. thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4091: HDFS-16517 Distance metric is wrong for non-DN machines in 2.10.
hadoop-yetus commented on pull request #4091: URL: https://github.com/apache/hadoop/pull/4091#issuecomment-1075521316 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 8m 27s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. | _ branch-2.10 Compile Tests _ | | +1 :green_heart: | mvninstall | 14m 47s | | branch-2.10 passed | | +1 :green_heart: | compile | 13m 52s | | branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | +1 :green_heart: | compile | 11m 25s | | branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 | | +1 :green_heart: | checkstyle | 0m 43s | | branch-2.10 passed | | +1 :green_heart: | mvnsite | 1m 20s | | branch-2.10 passed | | +1 :green_heart: | javadoc | 1m 16s | | branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | +1 :green_heart: | javadoc | 1m 7s | | branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 | | -1 :x: | spotbugs | 2m 6s | [/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/1/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html) | hadoop-common-project/hadoop-common in branch-2.10 has 2 extant spotbugs warnings. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 45s | | the patch passed | | +1 :green_heart: | compile | 13m 8s | | the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | +1 :green_heart: | javac | 13m 8s | | the patch passed | | +1 :green_heart: | compile | 11m 16s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 | | +1 :green_heart: | javac | 11m 16s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | +1 :green_heart: | checkstyle | 0m 41s | | the patch passed | | +1 :green_heart: | mvnsite | 1m 13s | | the patch passed | | +1 :green_heart: | javadoc | 1m 16s | | the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | +1 :green_heart: | javadoc | 1m 4s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 | | +1 :green_heart: | spotbugs | 2m 9s | | the patch passed | _ Other Tests _ | | -1 :x: | unit | 9m 20s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch passed. | | +1 :green_heart: | asflicense | 0m 49s | | The patch does not generate ASF License warnings. | | | | 101m 36s | | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.fs.sftp.TestSFTPFileSystem | | | hadoop.io.compress.snappy.TestSnappyCompressorDecompressor | | | hadoop.io.compress.TestCompressorDecompressor | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4091 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell | | uname | Linux 6317914f6ae6 4.15.0-161-generic #169-Ubuntu SMP Fri Oct 15 13:41:54 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | branch-2.10 / be14b0cad50fed2d8ecb42d236aed397bebc9609 | | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 | | Multi-JDK versions | /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_262-b10 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/1/testReport/ | | Max. process+thread count | 1397 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/1/console | | versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://ye
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #4074: HADOOP-18160 Avoid shading wildfly.openssl runtime dependency
hadoop-yetus removed a comment on pull request #4074: URL: https://github.com/apache/hadoop/pull/4074#issuecomment-1069297089 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18160) `org.wildfly.openssl` should not be shaded by Hadoop build
[ https://issues.apache.org/jira/browse/HADOOP-18160?focusedWorklogId=746044&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746044 ] ASF GitHub Bot logged work on HADOOP-18160: --- Author: ASF GitHub Bot Created on: 22/Mar/22 19:01 Start Date: 22/Mar/22 19:01 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #4074: URL: https://github.com/apache/hadoop/pull/4074#issuecomment-1069297089 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 746044) Time Spent: 1h 40m (was: 1.5h) > `org.wildfly.openssl` should not be shaded by Hadoop build > -- > > Key: HADOOP-18160 > URL: https://issues.apache.org/jira/browse/HADOOP-18160 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.3.1 > Environment: hadoop 3.3.1 > spark 3.2.1 > JDK8 >Reporter: André F. >Assignee: André F. >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.3 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > `org.wildfly.openssl` is a runtime library and its references are being > shaded on Hadoop, breaking the integration with other frameworks like Spark, > whenever the "fs.s3a.ssl.channel.mode" is set to "openssl". The error > produced in this situation is: > {code:java} > Suppressed: java.lang.NoClassDefFoundError: > org/apache/hadoop/shaded/org/wildfly/openssl/OpenSSLProvider{code} > Whenever it tries to be instantiated from the `DelegatingSSLSocketFactory`. > Spark tries to add it to its classpath without the shade, thus creating this > issue. > Dependencies which are not on "compile" scope should probably not be shaded > to avoid this kind of integration issues. > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18160) `org.wildfly.openssl` should not be shaded by Hadoop build
[ https://issues.apache.org/jira/browse/HADOOP-18160?focusedWorklogId=746042&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746042 ] ASF GitHub Bot logged work on HADOOP-18160: --- Author: ASF GitHub Bot Created on: 22/Mar/22 19:01 Start Date: 22/Mar/22 19:01 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #4074: URL: https://github.com/apache/hadoop/pull/4074#issuecomment-1075521131 ok, merged in to branch-3.3 and trunk. thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 746042) Time Spent: 1.5h (was: 1h 20m) > `org.wildfly.openssl` should not be shaded by Hadoop build > -- > > Key: HADOOP-18160 > URL: https://issues.apache.org/jira/browse/HADOOP-18160 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.3.1 > Environment: hadoop 3.3.1 > spark 3.2.1 > JDK8 >Reporter: André F. >Assignee: André F. >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.3 > > Time Spent: 1.5h > Remaining Estimate: 0h > > `org.wildfly.openssl` is a runtime library and its references are being > shaded on Hadoop, breaking the integration with other frameworks like Spark, > whenever the "fs.s3a.ssl.channel.mode" is set to "openssl". The error > produced in this situation is: > {code:java} > Suppressed: java.lang.NoClassDefFoundError: > org/apache/hadoop/shaded/org/wildfly/openssl/OpenSSLProvider{code} > Whenever it tries to be instantiated from the `DelegatingSSLSocketFactory`. > Spark tries to add it to its classpath without the shade, thus creating this > issue. > Dependencies which are not on "compile" scope should probably not be shaded > to avoid this kind of integration issues. > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4091: HDFS-16517 Distance metric is wrong for non-DN machines in 2.10.
hadoop-yetus commented on pull request #4091: URL: https://github.com/apache/hadoop/pull/4091#issuecomment-1075543846 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 12m 10s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. | _ branch-2.10 Compile Tests _ | | +1 :green_heart: | mvninstall | 15m 55s | | branch-2.10 passed | | +1 :green_heart: | compile | 14m 15s | | branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | +1 :green_heart: | compile | 13m 9s | | branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 | | +1 :green_heart: | checkstyle | 0m 36s | | branch-2.10 passed | | +1 :green_heart: | mvnsite | 1m 25s | | branch-2.10 passed | | +1 :green_heart: | javadoc | 1m 17s | | branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | +1 :green_heart: | javadoc | 1m 0s | | branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 | | -1 :x: | spotbugs | 2m 4s | [/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/2/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html) | hadoop-common-project/hadoop-common in branch-2.10 has 2 extant spotbugs warnings. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 50s | | the patch passed | | +1 :green_heart: | compile | 13m 52s | | the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | -1 :x: | javac | 13m 52s | [/results-compile-javac-root-jdkAzulSystems,Inc.-1.7.0_262-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/2/artifact/out/results-compile-javac-root-jdkAzulSystems,Inc.-1.7.0_262-b10.txt) | root-jdkAzulSystems,Inc.-1.7.0_262-b10 with JDK Azul Systems, Inc.-1.7.0_262-b10 generated 1 new + 1678 unchanged - 1 fixed = 1679 total (was 1679) | | +1 :green_heart: | compile | 11m 35s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 | | -1 :x: | javac | 11m 35s | [/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/2/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07.txt) | root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 generated 1 new + 1578 unchanged - 1 fixed = 1579 total (was 1579) | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | +1 :green_heart: | checkstyle | 0m 35s | | the patch passed | | +1 :green_heart: | mvnsite | 1m 17s | | the patch passed | | +1 :green_heart: | javadoc | 1m 12s | | the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 | | +1 :green_heart: | spotbugs | 2m 20s | | the patch passed | _ Other Tests _ | | -1 :x: | unit | 10m 29s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch passed. | | +1 :green_heart: | asflicense | 0m 44s | | The patch does not generate ASF License warnings. | | | | 110m 7s | | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.io.compress.TestCompressorDecompressor | | | hadoop.io.compress.snappy.TestSnappyCompressorDecompressor | | | hadoop.fs.sftp.TestSFTPFileSystem | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4091 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell | | uname | Linux a08702ad2c29 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | branch-2.10 / 1d7220b77
[GitHub] [hadoop] hadoop-yetus commented on pull request #4089: HDFS-16515. Improve ec exception message
hadoop-yetus commented on pull request #4089: URL: https://github.com/apache/hadoop/pull/4089#issuecomment-1075580681 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 1m 14s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 1s | | codespell was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 37m 6s | | trunk passed | | +1 :green_heart: | compile | 1m 31s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | compile | 1m 19s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | checkstyle | 1m 1s | | trunk passed | | +1 :green_heart: | mvnsite | 1m 29s | | trunk passed | | +1 :green_heart: | javadoc | 1m 4s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 1m 29s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 3m 29s | | trunk passed | | +1 :green_heart: | shadedclient | 26m 19s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 19s | | the patch passed | | +1 :green_heart: | compile | 1m 24s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javac | 1m 24s | | the patch passed | | +1 :green_heart: | compile | 1m 16s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | javac | 1m 16s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | +1 :green_heart: | checkstyle | 0m 53s | | the patch passed | | +1 :green_heart: | mvnsite | 1m 24s | | the patch passed | | +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 1m 23s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 3m 29s | | the patch passed | | +1 :green_heart: | shadedclient | 25m 55s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 373m 48s | | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 0m 38s | | The patch does not generate ASF License warnings. | | | | 485m 28s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4089/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4089 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell | | uname | Linux 02502b1e4a91 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 9073517068c58944279fa81478bb3f3cb6e21543 | | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4089/2/testReport/ | | Max. process+thread count | 2005 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4089/2/console | | versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hadoop] xinglin commented on pull request #4094: HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS
xinglin commented on pull request #4094: URL: https://github.com/apache/hadoop/pull/4094#issuecomment-1075602418 > Could you add a pointer to where/how the corresponding logic lives in the non-abstract filesystem object, just to easily verify the logic is the same The logic is in DelegationTokenIssuer.java for FileSystem.java/ViewFileSystem.java. `collectDelegationTokens()` in DelegationTokenIssuer will collect token for the current issuer and all AdditionalTokenIssuers. In FileSystem.java, we overwrite `getAdditionalTokenIssuers()` to call `getChildFileSystem()` and ViewFileSystem overwrites `getChildFileSystem()`, to include the fallback FS as well. So, ViewFileSystem.java is good here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18169) getDelegationTokens in ViewFs should also fetch the token from the fallback FS
[ https://issues.apache.org/jira/browse/HADOOP-18169?focusedWorklogId=746103&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746103 ] ASF GitHub Bot logged work on HADOOP-18169: --- Author: ASF GitHub Bot Created on: 22/Mar/22 20:23 Start Date: 22/Mar/22 20:23 Worklog Time Spent: 10m Work Description: xinglin commented on pull request #4094: URL: https://github.com/apache/hadoop/pull/4094#issuecomment-1075602418 > Could you add a pointer to where/how the corresponding logic lives in the non-abstract filesystem object, just to easily verify the logic is the same The logic is in DelegationTokenIssuer.java for FileSystem.java/ViewFileSystem.java. `collectDelegationTokens()` in DelegationTokenIssuer will collect token for the current issuer and all AdditionalTokenIssuers. In FileSystem.java, we overwrite `getAdditionalTokenIssuers()` to call `getChildFileSystem()` and ViewFileSystem overwrites `getChildFileSystem()`, to include the fallback FS as well. So, ViewFileSystem.java is good here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 746103) Time Spent: 0.5h (was: 20m) > getDelegationTokens in ViewFs should also fetch the token from the fallback FS > -- > > Key: HADOOP-18169 > URL: https://issues.apache.org/jira/browse/HADOOP-18169 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xing Lin >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > getDelegationTokens in ViewFs does not include the delegationToken from the > fallback FS, while it should. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4093: YARN-11093: fix for reading files in timeline server in chronological order in case of fs-support-append to false
hadoop-yetus commented on pull request #4093: URL: https://github.com/apache/hadoop/pull/4093#issuecomment-1075605670 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 59s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 2s | | codespell was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | _ trunk Compile Tests _ | | +0 :ok: | mvndep | 12m 45s | | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 27m 29s | | trunk passed | | +1 :green_heart: | compile | 12m 1s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | compile | 9m 34s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | checkstyle | 1m 44s | | trunk passed | | +1 :green_heart: | mvnsite | 1m 28s | | trunk passed | | +1 :green_heart: | javadoc | 1m 18s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 2m 52s | | trunk passed | | +1 :green_heart: | shadedclient | 24m 18s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 1s | | the patch passed | | +1 :green_heart: | compile | 11m 35s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javac | 11m 35s | | the patch passed | | +1 :green_heart: | compile | 10m 39s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | javac | 10m 39s | | the patch passed | | -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4093/1/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | -0 :warning: | checkstyle | 1m 51s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4093/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt) | hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 170 unchanged - 0 fixed = 171 total (was 170) | | +1 :green_heart: | mvnsite | 1m 35s | | the patch passed | | +1 :green_heart: | javadoc | 1m 24s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 1m 7s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 3m 29s | | the patch passed | | +1 :green_heart: | shadedclient | 24m 24s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 1m 0s | | hadoop-yarn-api in the patch passed. | | -1 :x: | unit | 11m 59s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4093/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt) | hadoop-yarn-server-timeline-pluginstorage in the patch passed. | | +1 :green_heart: | asflicense | 0m 46s | | The patch does not generate ASF License warnings. | | | | 169m 58s | | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4093/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4093 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell | | uname | Linux a37fe22eaa11 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven |
[GitHub] [hadoop] hadoop-yetus commented on pull request #4092: HADOOP-18167. Add metrics to track delegation token secret manager op…
hadoop-yetus commented on pull request #4092: URL: https://github.com/apache/hadoop/pull/4092#issuecomment-1075642781 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 17m 22s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 1s | | codespell was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 36m 34s | | trunk passed | | +1 :green_heart: | compile | 24m 39s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | compile | 21m 6s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | checkstyle | 1m 0s | | trunk passed | | +1 :green_heart: | mvnsite | 1m 37s | | trunk passed | | +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 1m 42s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 2m 31s | | trunk passed | | +1 :green_heart: | shadedclient | 25m 10s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 0s | | the patch passed | | +1 :green_heart: | compile | 23m 59s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javac | 23m 59s | | the patch passed | | +1 :green_heart: | compile | 21m 3s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | javac | 21m 3s | | the patch passed | | -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4092/1/artifact/out/blanks-eol.txt) | The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | -0 :warning: | checkstyle | 1m 0s | [/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4092/1/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt) | hadoop-common-project/hadoop-common: The patch generated 8 new + 74 unchanged - 0 fixed = 82 total (was 74) | | +1 :green_heart: | mvnsite | 1m 36s | | the patch passed | | +1 :green_heart: | javadoc | 1m 4s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 1m 38s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 2m 40s | | the patch passed | | +1 :green_heart: | shadedclient | 25m 43s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 17m 45s | | hadoop-common in the patch passed. | | +1 :green_heart: | asflicense | 0m 47s | | The patch does not generate ASF License warnings. | | | | 230m 29s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4092/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4092 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell | | uname | Linux 8a53c3388f5c 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 373b7db473260e8373a0ec8a12a57ea85e296269 | | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4092/1/testReport/ | | Max. process+thread count | 1237 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4092/1/console | | versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Y
[jira] [Work logged] (HADOOP-18167) Add metrics to track delegation token secret manager operations
[ https://issues.apache.org/jira/browse/HADOOP-18167?focusedWorklogId=746153&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746153 ] ASF GitHub Bot logged work on HADOOP-18167: --- Author: ASF GitHub Bot Created on: 22/Mar/22 21:10 Start Date: 22/Mar/22 21:10 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #4092: URL: https://github.com/apache/hadoop/pull/4092#issuecomment-1075642781 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 17m 22s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 1s | | codespell was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 36m 34s | | trunk passed | | +1 :green_heart: | compile | 24m 39s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | compile | 21m 6s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | checkstyle | 1m 0s | | trunk passed | | +1 :green_heart: | mvnsite | 1m 37s | | trunk passed | | +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 1m 42s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 2m 31s | | trunk passed | | +1 :green_heart: | shadedclient | 25m 10s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 0s | | the patch passed | | +1 :green_heart: | compile | 23m 59s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javac | 23m 59s | | the patch passed | | +1 :green_heart: | compile | 21m 3s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | javac | 21m 3s | | the patch passed | | -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4092/1/artifact/out/blanks-eol.txt) | The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | -0 :warning: | checkstyle | 1m 0s | [/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4092/1/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt) | hadoop-common-project/hadoop-common: The patch generated 8 new + 74 unchanged - 0 fixed = 82 total (was 74) | | +1 :green_heart: | mvnsite | 1m 36s | | the patch passed | | +1 :green_heart: | javadoc | 1m 4s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 1m 38s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 2m 40s | | the patch passed | | +1 :green_heart: | shadedclient | 25m 43s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 17m 45s | | hadoop-common in the patch passed. | | +1 :green_heart: | asflicense | 0m 47s | | The patch does not generate ASF License warnings. | | | | 230m 29s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4092/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4092 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell | | uname | Linux 8a53c3388f5c 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 373b7db473260e8373a0ec8a12a57ea85e296269 | | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | T
[jira] [Commented] (HADOOP-18167) Add metrics to track delegation token secret manager operations
[ https://issues.apache.org/jira/browse/HADOOP-18167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510910#comment-17510910 ] Hector Sandoval Chaverri commented on HADOOP-18167: --- Hi [~ste...@apache.org], I saw that a few classes , such as S3AInstrumentation, implement IOStatisticsSource and use IOStatisticsStore to track different counters. Is this the approach that you think we should follow? Could you also help explain what's the consumer of the IOStatistics? > Add metrics to track delegation token secret manager operations > --- > > Key: HADOOP-18167 > URL: https://issues.apache.org/jira/browse/HADOOP-18167 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > New metrics to track operations that store, update and remove delegation > tokens in implementations of AbstractDelegationTokenSecretManager. This will > help evaluate the impact of using different secret managers and add > optimizations. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] abhishekdas99 commented on a change in pull request #4094: HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS
abhishekdas99 commented on a change in pull request #4094: URL: https://github.com/apache/hadoop/pull/4094#discussion_r832668185 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsLinkFallback.java ## @@ -182,6 +184,26 @@ public void testMkdirOfNewDirWithOutMatchingToMountOrFallbackDirTree() assertTrue(fsTarget.exists(test)); } + /** + * Test getDelegationToken when fallback is configured + */ + @Test + public void testGetDelegationToken() throws IOException { Review comment: If we are specifically testing for fallback, can you please change the name to include fallback? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18169) getDelegationTokens in ViewFs should also fetch the token from the fallback FS
[ https://issues.apache.org/jira/browse/HADOOP-18169?focusedWorklogId=746204&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746204 ] ASF GitHub Bot logged work on HADOOP-18169: --- Author: ASF GitHub Bot Created on: 22/Mar/22 22:08 Start Date: 22/Mar/22 22:08 Worklog Time Spent: 10m Work Description: abhishekdas99 commented on a change in pull request #4094: URL: https://github.com/apache/hadoop/pull/4094#discussion_r832668185 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsLinkFallback.java ## @@ -182,6 +184,26 @@ public void testMkdirOfNewDirWithOutMatchingToMountOrFallbackDirTree() assertTrue(fsTarget.exists(test)); } + /** + * Test getDelegationToken when fallback is configured + */ + @Test + public void testGetDelegationToken() throws IOException { Review comment: If we are specifically testing for fallback, can you please change the name to include fallback? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 746204) Time Spent: 40m (was: 0.5h) > getDelegationTokens in ViewFs should also fetch the token from the fallback FS > -- > > Key: HADOOP-18169 > URL: https://issues.apache.org/jira/browse/HADOOP-18169 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xing Lin >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > getDelegationTokens in ViewFs does not include the delegationToken from the > fallback FS, while it should. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4091: HDFS-16517 Distance metric is wrong for non-DN machines in 2.10.
hadoop-yetus commented on pull request #4091: URL: https://github.com/apache/hadoop/pull/4091#issuecomment-1075730771 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 40s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. | _ branch-2.10 Compile Tests _ | | +1 :green_heart: | mvninstall | 14m 11s | | branch-2.10 passed | | +1 :green_heart: | compile | 12m 56s | | branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | +1 :green_heart: | compile | 10m 44s | | branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 | | +1 :green_heart: | checkstyle | 0m 42s | | branch-2.10 passed | | +1 :green_heart: | mvnsite | 1m 19s | | branch-2.10 passed | | +1 :green_heart: | javadoc | 1m 16s | | branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | +1 :green_heart: | javadoc | 1m 4s | | branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 | | -1 :x: | spotbugs | 2m 1s | [/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/3/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html) | hadoop-common-project/hadoop-common in branch-2.10 has 2 extant spotbugs warnings. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 44s | | the patch passed | | +1 :green_heart: | compile | 12m 21s | | the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | +1 :green_heart: | javac | 12m 21s | | the patch passed | | +1 :green_heart: | compile | 10m 37s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 | | +1 :green_heart: | javac | 10m 37s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | +1 :green_heart: | checkstyle | 0m 40s | | the patch passed | | +1 :green_heart: | mvnsite | 1m 15s | | the patch passed | | +1 :green_heart: | javadoc | 1m 15s | | the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | +1 :green_heart: | javadoc | 1m 2s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 | | +1 :green_heart: | spotbugs | 2m 10s | | the patch passed | _ Other Tests _ | | -1 :x: | unit | 9m 12s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/3/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch passed. | | +1 :green_heart: | asflicense | 0m 50s | | The patch does not generate ASF License warnings. | | | | 89m 53s | | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.fs.sftp.TestSFTPFileSystem | | | hadoop.io.compress.snappy.TestSnappyCompressorDecompressor | | | hadoop.util.TestDiskCheckerWithDiskIo | | | hadoop.io.compress.TestCompressorDecompressor | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4091 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell | | uname | Linux 6b99d20a8120 4.15.0-161-generic #169-Ubuntu SMP Fri Oct 15 13:41:54 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | branch-2.10 / 1326ef2a4f0d22c365603144812fac73dfa1f962 | | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 | | Multi-JDK versions | /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_262-b10 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/3/testReport/ | | Max. process+thread count | 1393 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/3/console | | versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 | | Pow
[GitHub] [hadoop] xinglin commented on a change in pull request #4091: HDFS-16517 Distance metric is wrong for non-DN machines in 2.10.
xinglin commented on a change in pull request #4091: URL: https://github.com/apache/hadoop/pull/4091#discussion_r832726638 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java ## @@ -823,7 +823,7 @@ private static int getWeightUsingNetworkLocation(Node reader, Node node) { } currentLevel++; } -weight = (readerPathToken.length - currentLevel) + +weight = 2 + (readerPathToken.length - currentLevel) + Review comment: nit: could we move "2 + " to the end of the next line, so that it is identical to what is in trunk? ``` weight = (readerPathToken.length - currentLevel) + (nodePathToken.length - currentLevel) + 2; ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xinglin commented on a change in pull request #4089: HDFS-16515. Improve ec exception message
xinglin commented on a change in pull request #4089: URL: https://github.com/apache/hadoop/pull/4089#discussion_r832768475 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirErasureCodingOp.java ## @@ -168,8 +168,8 @@ static FileStatus setErasureCodingPolicy(final FSNamesystem fsn, throw new FileNotFoundException("Path not found: " + srcIIP.getPath()); } if (!inode.isDirectory()) { - throw new IOException("Attempt to set an erasure coding policy " + - "for a file " + src); + throw new IOException("Cannot set an erasure coding policy " + + "for a file " + src + ". Expected to be a directory"); Review comment: nit: "Expected to be a directory" -> "Erasure coding policy can only be set for directories." -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4094: HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS
hadoop-yetus commented on pull request #4094: URL: https://github.com/apache/hadoop/pull/4094#issuecomment-1075837818 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 12m 22s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. | _ trunk Compile Tests _ | | +0 :ok: | mvndep | 12m 41s | | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 23m 25s | | trunk passed | | +1 :green_heart: | compile | 24m 12s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | compile | 21m 22s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | checkstyle | 3m 46s | | trunk passed | | +1 :green_heart: | mvnsite | 3m 23s | | trunk passed | | +1 :green_heart: | javadoc | 2m 22s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 3m 30s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 6m 15s | | trunk passed | | +1 :green_heart: | shadedclient | 24m 56s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 22s | | the patch passed | | +1 :green_heart: | compile | 23m 42s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javac | 23m 42s | | the patch passed | | +1 :green_heart: | compile | 22m 0s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | javac | 22m 0s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | -0 :warning: | checkstyle | 3m 35s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/1/artifact/out/results-checkstyle-root.txt) | root: The patch generated 1 new + 41 unchanged - 0 fixed = 42 total (was 41) | | +1 :green_heart: | mvnsite | 3m 20s | | the patch passed | | +1 :green_heart: | javadoc | 2m 20s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 3m 23s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 7m 4s | | the patch passed | | +1 :green_heart: | shadedclient | 25m 24s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 18m 18s | | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 242m 39s | | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 1m 9s | | The patch does not generate ASF License warnings. | | | | 492m 7s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4094 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell | | uname | Linux 716989d0f40f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / f5adca1a0ed64818f09cce3082dcd887ad03102c | | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/1/testReport/ | | Max. process+thread count | 3534 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/1/console | | versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apach