Re: [PR] [ABFS] [Backport 3.4] Back Merging PRs from trunk to Branch 3.4 [hadoop]
hadoop-yetus commented on PR #6611:
URL: https://github.com/apache/hadoop/pull/6611#issuecomment-1978079289
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 23s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 18 new or modified test files. |
_ branch-3.4 Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 38s | | branch-3.4 passed |
| +1 :green_heart: | compile | 0m 19s | | branch-3.4 passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 20s | | branch-3.4 passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 22s | | branch-3.4 passed |
| +1 :green_heart: | mvnsite | 0m 28s | | branch-3.4 passed |
| +1 :green_heart: | javadoc | 0m 26s | | branch-3.4 passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 22s | | branch-3.4 passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 45s | | branch-3.4 passed |
| +1 :green_heart: | shadedclient | 19m 37s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 0m 19s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 0m 14s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 13s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6611/1/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 2 new + 10 unchanged - 0
fixed = 12 total (was 10) |
| +1 :green_heart: | mvnsite | 0m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 41s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 16s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 51s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 24s | | The patch does not
generate ASF License warnings. |
| | | 82m 19s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6611/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6611 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux b238ccd57596 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.4 / c9cd6138450b1fe12640025b903aac8840a91551 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6611/1/testReport/ |
| Max. process+thread count | 551 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6611/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.
Re: [PR] HDFS-17396. BootstrapStandby should download rollback image during RollingUpgrade [hadoop]
hadoop-yetus commented on PR #6583:
URL: https://github.com/apache/hadoop/pull/6583#issuecomment-1978026417
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 19s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 59s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 12s | | trunk passed |
| +1 :green_heart: | compile | 2m 53s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 2m 44s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 43s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 5s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 3s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 21s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 0m 42s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6583/7/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf-warnings.html)
| hadoop-hdfs-project/hadoop-hdfs-rbf in trunk has 1 extant spotbugs
warnings. |
| +1 :green_heart: | shadedclient | 19m 23s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 52s | | the patch passed |
| +1 :green_heart: | compile | 2m 48s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | cc | 2m 48s | | the patch passed |
| +1 :green_heart: | javac | 2m 48s | | the patch passed |
| +1 :green_heart: | compile | 2m 39s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | cc | 2m 39s | | the patch passed |
| +1 :green_heart: | javac | 2m 39s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 34s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 56s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 16s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 37s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 44s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 198m 45s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6583/7/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 12m 36s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 314m 41s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.tools.TestDFSAdmin |
| | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| | hadoop.hdfs.protocol.TestBlockListAsLongs |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6583/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6583 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint
bufcompat |
| uname | Linux 3f94366f8e4c 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop
Re: [PR] HADOOP-18980. Invalid inputs for getTrimmedStringCollectionSplitByEquals (ADDENDUM) [hadoop]
virajjasani commented on PR #6546: URL: https://github.com/apache/hadoop/pull/6546#issuecomment-1978003122 @steveloughran a gentle reminder :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] [ABFS] [Backport 3.4] Back Merging PRs from trunk to Branch 3.4 [hadoop]
anujmodi2021 opened a new pull request, #6611: URL: https://github.com/apache/hadoop/pull/6611 ### Description of PR Following two PRs were merged to trunk. #5881 #6069 These two PRs may have conflicts thus creating this single backport PR to avoid any merge issues. Commits Backported from trunk: https://github.com/apache/hadoop/commit/1336c362e5b685c020b667e308f97f7bdb5b6fd8 https://github.com/apache/hadoop/commit/99b9e7fb43b596f67983b5a8a8859d9efc886341 ### How was this patch tested? Both the commits were cherry-picked and ABFS Test Suite was run. ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] Test build [hadoop]
hadoop-yetus commented on PR #6610:
URL: https://github.com/apache/hadoop/pull/6610#issuecomment-1977963439
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 33s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 44s | | trunk passed |
| +1 :green_heart: | compile | 2m 55s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 2m 48s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 46s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 19s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 5s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 1m 26s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6610/1/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html)
| hadoop-hdfs-project/hadoop-hdfs-client in trunk has 1 extant spotbugs
warnings. |
| +1 :green_heart: | shadedclient | 20m 35s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 20m 48s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 6s | | the patch passed |
| +1 :green_heart: | compile | 2m 53s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 2m 53s | | the patch passed |
| +1 :green_heart: | compile | 2m 43s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 2m 43s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6610/1/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | checkstyle | 0m 37s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 7s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 54s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 30s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 51s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 198m 19s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6610/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not
generate ASF License warnings. |
| | | 305m 35s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| | hadoop.hdfs.protocol.TestBlockListAsLongs |
| | hadoop.hdfs.tools.TestDFSAdmin |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6610/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6610 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 7a4febdd345b 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 U
Re: [PR] HDFS-17398. [FGL] Implement the FGL lock for FSNLockManager [hadoop]
ferhui commented on code in PR #6599:
URL: https://github.com/apache/hadoop/pull/6599#discussion_r1512099828
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/fgl/FineGrainedFSNamesystemLock.java:
##
@@ -0,0 +1,293 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.namenode.fgl;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock;
+import org.apache.hadoop.metrics2.lib.MutableRatesWithAggregation;
+
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.Supplier;
+
+/**
+ * Splitting the global FSN lock into FSLock and BMLock.
+ * FSLock is used to protect directory tree-related operations.
+ * BMLock is used to protect block-related and dn-related operations.
+ * The lock order should be: FSLock -> BMLock.
+ */
+public class FineGrainedFSNamesystemLock implements FSNLockManager {
+ private final FSNamesystemLock fsLock;
+ private final FSNamesystemLock bmLock;
+
+ public FineGrainedFSNamesystemLock(Configuration conf,
MutableRatesWithAggregation aggregation) {
+this.fsLock = new FSNamesystemLock(conf, aggregation);
+this.bmLock = new FSNamesystemLock(conf, aggregation);
+ }
+
+ @Override
+ public void readLock(FSNamesystemLockMode lockMode) {
+if (lockMode.equals(FSNamesystemLockMode.GLOBAL)) {
+ this.fsLock.readLock();
+ this.bmLock.readLock();
+} else if (lockMode.equals(FSNamesystemLockMode.FS)) {
+ this.fsLock.readLock();
+} else if (lockMode.equals(FSNamesystemLockMode.BM)) {
+ this.bmLock.readLock();
+}
+ }
+
+ public void readLockInterruptibly(FSNamesystemLockMode lockMode) throws
InterruptedException {
+if (lockMode.equals(FSNamesystemLockMode.GLOBAL)) {
+ this.fsLock.readLockInterruptibly();
+ try {
+this.bmLock.readLockInterruptibly();
+ } catch (InterruptedException e) {
+// release the holding FSLock
+this.fsLock.readUnlock("BMReadLockInterruptiblyFailed");
+throw e;
+ }
+} else if (lockMode.equals(FSNamesystemLockMode.FS)) {
+ this.fsLock.readLockInterruptibly();
+} else if (lockMode.equals(FSNamesystemLockMode.BM)) {
+ this.bmLock.readLockInterruptibly();
+}
+ }
+
+ @Override
+ public void readUnlock(FSNamesystemLockMode lockMode, String opName) {
+if (lockMode.equals(FSNamesystemLockMode.GLOBAL)) {
+ this.bmLock.readUnlock(opName);
+ this.fsLock.readUnlock(opName);
+} else if (lockMode.equals(FSNamesystemLockMode.FS)) {
+ this.fsLock.readUnlock(opName);
+} else if (lockMode.equals(FSNamesystemLockMode.BM)) {
+ this.bmLock.readUnlock(opName);
+}
+ }
+
+ public void readUnlock(FSNamesystemLockMode lockMode, String opName,
+ Supplier lockReportInfoSupplier) {
+if (lockMode.equals(FSNamesystemLockMode.GLOBAL)) {
+ this.bmLock.readUnlock(opName, lockReportInfoSupplier);
+ this.fsLock.readUnlock(opName, lockReportInfoSupplier);
+} else if (lockMode.equals(FSNamesystemLockMode.FS)) {
+ this.fsLock.readUnlock(opName, lockReportInfoSupplier);
+} else if (lockMode.equals(FSNamesystemLockMode.BM)) {
+ this.bmLock.readUnlock(opName, lockReportInfoSupplier);
+}
+ }
+
+ @Override
+ public void writeLock(FSNamesystemLockMode lockMode) {
+if (lockMode.equals(FSNamesystemLockMode.GLOBAL)) {
+ this.fsLock.writeLock();
+ this.bmLock.writeLock();
+} else if (lockMode.equals(FSNamesystemLockMode.FS)) {
+ this.fsLock.writeLock();
+} else if (lockMode.equals(FSNamesystemLockMode.BM)) {
+ this.bmLock.writeLock();
+}
+ }
+
+ @Override
+ public void writeUnlock(FSNamesystemLockMode lockMode, String opName) {
+if (lockMode.equals(FSNamesystemLockMode.GLOBAL)) {
+ this.bmLock.writeUnlock(opName);
+ this.fsLock.writeUnlock(opName);
+} else if (lockMode.equals(FSNamesystemLockMode.FS)) {
+ this.fsLock.writeUnlock(opName);
+} else if (lockMode.equals(FSNamesystemLockMode.BM)) {
+ this.bmLock.writeUnlock(opName);
+}
+ }
+
+ @Override
+ public void writeUnlock(FSNamesystemLockMode lockMo
Re: [PR] HDFS-17299. Adding rack failure tolerance when creating a new file [hadoop]
hadoop-yetus commented on PR #6566:
URL: https://github.com/apache/hadoop/pull/6566#issuecomment-1977869072
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 2s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 5s | | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 0m 25s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/18/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 0m 25s |
[/branch-compile-hadoop-hdfs-project-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/18/artifact/out/branch-compile-hadoop-hdfs-project-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| hadoop-hdfs-project in trunk failed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | compile | 0m 23s |
[/branch-compile-hadoop-hdfs-project-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/18/artifact/out/branch-compile-hadoop-hdfs-project-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| hadoop-hdfs-project in trunk failed with JDK Private
Build-1.8.0_392-8u392-ga-1~20.04-b08. |
| -0 :warning: | checkstyle | 0m 21s |
[/buildtool-branch-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/18/artifact/out/buildtool-branch-checkstyle-hadoop-hdfs-project.txt)
| The patch fails to run checkstyle in hadoop-hdfs-project |
| -1 :x: | mvnsite | 0m 23s |
[/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/18/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-client.txt)
| hadoop-hdfs-client in trunk failed. |
| -1 :x: | mvnsite | 0m 23s |
[/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/18/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in trunk failed. |
| -1 :x: | javadoc | 0m 21s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/18/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| hadoop-hdfs-client in trunk failed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | javadoc | 0m 22s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/18/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| hadoop-hdfs in trunk failed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | javadoc | 0m 23s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/18/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| hadoop-hdfs-client in trunk failed with JDK Private
Build-1.8.0_392-8u392-ga-1~20.04-b08. |
| -1 :x: | javadoc | 0m 22s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/18/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| hadoop-hdfs in trunk failed with JDK Private
Build-1.8.0_392-8u392-ga-1~20.04-b08. |
| -1 :x: | spotbugs | 0m 23s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/18/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.txt)
| hadoop-hdfs-client in trunk failed. |
| -1 :x: | spotbugs | 0m 23s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/18/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in tru
Re: [PR] HDFS-17404. Add NN Socket Address to log when processing command from active NN [hadoop]
hadoop-yetus commented on PR #6609:
URL: https://github.com/apache/hadoop/pull/6609#issuecomment-1977694056
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 5m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 22s | | branch-3.3 passed |
| +1 :green_heart: | compile | 1m 17s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 51s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 1m 29s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 1m 34s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 3m 18s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 37m 22s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 1m 11s | | the patch passed |
| +1 :green_heart: | javac | 1m 11s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 41s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 19s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 26s | | the patch passed |
| +1 :green_heart: | spotbugs | 3m 16s | | the patch passed |
| +1 :green_heart: | shadedclient | 37m 5s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 195m 4s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6609/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 340m 32s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.TestFileTruncate |
| | hadoop.hdfs.protocol.TestBlockListAsLongs |
| | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6609/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6609 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 1fe9283b12ee 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / 84112ba5298b392dccd78569ac59eb2e49228821 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6609/1/testReport/ |
| Max. process+thread count | 2993 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6609/1/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] Test build [hadoop]
ritegarg opened a new pull request, #6610: URL: https://github.com/apache/hadoop/pull/6610 ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17299. Adding rack failure tolerance when creating a new file [hadoop]
hadoop-yetus commented on PR #6566:
URL: https://github.com/apache/hadoop/pull/6566#issuecomment-1977657875
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 4m 2s | | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 0m 34s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/16/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 2m 25s |
[/branch-compile-hadoop-hdfs-project-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/16/artifact/out/branch-compile-hadoop-hdfs-project-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| hadoop-hdfs-project in trunk failed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | compile | 0m 23s |
[/branch-compile-hadoop-hdfs-project-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/16/artifact/out/branch-compile-hadoop-hdfs-project-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| hadoop-hdfs-project in trunk failed with JDK Private
Build-1.8.0_392-8u392-ga-1~20.04-b08. |
| -0 :warning: | checkstyle | 0m 21s |
[/buildtool-branch-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/16/artifact/out/buildtool-branch-checkstyle-hadoop-hdfs-project.txt)
| The patch fails to run checkstyle in hadoop-hdfs-project |
| -1 :x: | mvnsite | 0m 59s |
[/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/16/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in trunk failed. |
| -1 :x: | mvnsite | 0m 24s |
[/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/16/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-client.txt)
| hadoop-hdfs-client in trunk failed. |
| -1 :x: | javadoc | 0m 40s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/16/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| hadoop-hdfs in trunk failed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | javadoc | 0m 23s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/16/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| hadoop-hdfs-client in trunk failed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | javadoc | 0m 24s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/16/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| hadoop-hdfs in trunk failed with JDK Private
Build-1.8.0_392-8u392-ga-1~20.04-b08. |
| -1 :x: | spotbugs | 0m 14s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/16/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in trunk failed. |
| -1 :x: | spotbugs | 3m 32s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/16/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html)
| hadoop-hdfs-project/hadoop-hdfs-client in trunk has 1 extant spotbugs
warnings. |
| -1 :x: | shadedclient | 4m 8s | | branch has errors when building
and testing our client artifacts. |
| -0 :warning: | patch | 4m 32s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: |
[jira] [Commented] (HADOOP-19098) Vector IO: consistent specified rejection of overlapping ranges
[
https://issues.apache.org/jira/browse/HADOOP-19098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17823376#comment-17823376
]
ASF GitHub Bot commented on HADOOP-19098:
-
hadoop-yetus commented on PR #6604:
URL: https://github.com/apache/hadoop/pull/6604#issuecomment-1977653680
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 24s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 17s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 43s | | trunk passed |
| +1 :green_heart: | compile | 17m 17s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 16m 7s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 20s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 42s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 2m 27s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 2m 37s |
[/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6604/2/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html)
| hadoop-common-project/hadoop-common in trunk has 1 extant spotbugs
warnings. |
| +1 :green_heart: | shadedclient | 33m 20s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 21s |
[/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6604/2/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| -1 :x: | compile | 15m 56s |
[/patch-compile-root-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6604/2/artifact/out/patch-compile-root-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| root in the patch failed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | javac | 15m 56s |
[/patch-compile-root-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6604/2/artifact/out/patch-compile-root-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| root in the patch failed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | compile | 15m 30s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6604/2/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| root in the patch failed with JDK Private
Build-1.8.0_392-8u392-ga-1~20.04-b08. |
| -1 :x: | javac | 15m 30s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6604/2/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| root in the patch failed with JDK Private
Build-1.8.0_392-8u392-ga-1~20.04-b08. |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6604/2/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 4m 19s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6604/2/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 2 new + 56 unchanged - 0 fixed = 58 total (was
56) |
| -1 :x: | mvnsite | 0m 46s |
[/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6604/2/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch
Re: [PR] HADOOP-19098 Vector IO: consistent specified rejection of overlapping ranges [hadoop]
hadoop-yetus commented on PR #6604:
URL: https://github.com/apache/hadoop/pull/6604#issuecomment-1977653680
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 24s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 17s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 43s | | trunk passed |
| +1 :green_heart: | compile | 17m 17s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 16m 7s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 20s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 42s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 2m 27s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 2m 37s |
[/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6604/2/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html)
| hadoop-common-project/hadoop-common in trunk has 1 extant spotbugs
warnings. |
| +1 :green_heart: | shadedclient | 33m 20s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 21s |
[/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6604/2/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| -1 :x: | compile | 15m 56s |
[/patch-compile-root-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6604/2/artifact/out/patch-compile-root-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| root in the patch failed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | javac | 15m 56s |
[/patch-compile-root-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6604/2/artifact/out/patch-compile-root-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| root in the patch failed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | compile | 15m 30s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6604/2/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| root in the patch failed with JDK Private
Build-1.8.0_392-8u392-ga-1~20.04-b08. |
| -1 :x: | javac | 15m 30s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6604/2/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| root in the patch failed with JDK Private
Build-1.8.0_392-8u392-ga-1~20.04-b08. |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6604/2/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 4m 19s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6604/2/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 2 new + 56 unchanged - 0 fixed = 58 total (was
56) |
| -1 :x: | mvnsite | 0m 46s |
[/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6604/2/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| -1 :x: | javadoc | 1m 10s |
[/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6604/2/artifact/out/patch-javadoc-hadoop-common
Re: [PR] HDFS-17299. Adding rack failure tolerance when creating a new file [hadoop]
hadoop-yetus commented on PR #6566:
URL: https://github.com/apache/hadoop/pull/6566#issuecomment-1977608476
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 0s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 6s | | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 42m 15s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/15/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 2m 39s |
[/branch-compile-hadoop-hdfs-project-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/15/artifact/out/branch-compile-hadoop-hdfs-project-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| hadoop-hdfs-project in trunk failed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | compile | 8m 32s |
[/branch-compile-hadoop-hdfs-project-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/15/artifact/out/branch-compile-hadoop-hdfs-project-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| hadoop-hdfs-project in trunk failed with JDK Private
Build-1.8.0_392-8u392-ga-1~20.04-b08. |
| +1 :green_heart: | checkstyle | 2m 13s | | trunk passed |
| -1 :x: | mvnsite | 0m 30s |
[/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/15/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in trunk failed. |
| -1 :x: | javadoc | 0m 31s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/15/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| hadoop-hdfs-client in trunk failed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | javadoc | 0m 30s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/15/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| hadoop-hdfs in trunk failed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | javadoc | 0m 33s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/15/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| hadoop-hdfs in trunk failed with JDK Private
Build-1.8.0_392-8u392-ga-1~20.04-b08. |
| -1 :x: | spotbugs | 0m 27s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/15/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.txt)
| hadoop-hdfs-client in trunk failed. |
| -1 :x: | spotbugs | 0m 24s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/15/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in trunk failed. |
| +1 :green_heart: | shadedclient | 8m 39s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 9m 7s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 17s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 16s |
[/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/15/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-client.txt)
| hadoop-hdfs-client in the patch failed. |
| -1 :x: | compile | 0m 26s |
[/patch-compile-hadoop-hdfs-project-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/15/artifact/ou
[jira] [Updated] (HADOOP-19018) Release Hadoop 3.4.0
[
https://issues.apache.org/jira/browse/HADOOP-19018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HADOOP-19018:
Labels: pull-request-available (was: )
> Release Hadoop 3.4.0
>
>
> Key: HADOOP-19018
> URL: https://issues.apache.org/jira/browse/HADOOP-19018
> Project: Hadoop Common
> Issue Type: Task
>Affects Versions: 3.4.0
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Major
> Labels: pull-request-available
>
> Confirmed features to be included in the release:
> - Enhanced functionality for YARN Federation.
> - Redesigned resource allocation in YARN Capacity Scheduler
> - Optimization of HDFS RBF.
> - Introduction of fine-grained global locks for DataNodes.
> - Improvements in the stability of HDFS EC, and more.
> - Fixes for important CVEs.
> *Issues that need to be addressed in hadoop-3.4.0-RC0 version.*
> 1. confirm the JIRA target version/fix version is 3.4.0 to ensure that the
> version setting is correct.
> 2. confirm the highlight of hadoop-3.4.0.
> 3. backport branch-3.4.0/branch-3.4.
> {code:java}
> HADOOP-19040. mvn site commands fails due to MetricsSystem And
> MetricsSystemImpl changes.
> YARN-11634. [Addendum] Speed-up TestTimelineClient.
> MAPREDUCE-7468. [Addendum] Fix TestMapReduceChildJVM unit tests.
>
> Revert HDFS-16016. BPServiceActor to provide new thread to handle IBR.
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18583) hadoop checknative fails to load openssl 3.x
[
https://issues.apache.org/jira/browse/HADOOP-18583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17823345#comment-17823345
]
ASF GitHub Bot commented on HADOOP-18583:
-
steveloughran commented on PR #5256:
URL: https://github.com/apache/hadoop/pull/5256#issuecomment-1977458743
> hadoop-yarn-server-nodemanager's container-executor
no idea. best to discuss on yarn-dev
regarding crypton and checknative, we may to work on that. IMO checknative
should look for openssl, at least if we add a -openssl argument.
I don't think we have specific openssl issues. In #6425 I had to add error
string matching for openssl 1 messages indicating stale https connections
(these were surfacing deep in the AWS error stack). For 3.x it'd be good to
know that these strings were the same -or update them. Maybe you can create an
uber-jira "support openssl 3"
one more thing: what is the openssl FIPS story? as for strict fips support
we don't just want to talk to fips endpoints, we want to run on hosts where the
ssl lib doesn't have the untrusted algorithms at all.
> hadoop checknative fails to load openssl 3.x
>
>
> Key: HADOOP-18583
> URL: https://issues.apache.org/jira/browse/HADOOP-18583
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
>Affects Versions: 3.3.4
>Reporter: Sebastian Klemke
>Priority: Major
> Labels: pull-request-available
> Attachments: 100-hadoop-3.3.4-openssl-3.patch
>
>
> After building Hadoop 3.3.4 from source on Ubuntu 22.04, `hadoop checknative`
> reports
> {code:java}
> $ hadoop checknative
> 2022-12-21 22:12:02,106 INFO bzip2.Bzip2Factory: Successfully loaded &
> initialized native-bzip2 library system-native
> 2022-12-21 22:12:02,107 INFO zlib.ZlibFactory: Successfully loaded &
> initialized native-zlib library
> 2022-12-21 22:12:02,130 INFO nativeio.NativeIO: The native code was built
> without PMDK support.
> Native library checking:
> hadoop: true /hadoop/lib/native/libhadoop.so.1.0.0
> zlib: true /lib/x86_64-linux-gnu/libz.so.1
> zstd : true /lib/x86_64-linux-gnu/libzstd.so.1
> bzip2: true /lib/x86_64-linux-gnu/libbz2.so.1
> openssl: false EVP_CIPHER_CTX_block_size
> ISA-L: true /lib/x86_64-linux-gnu/libisal.so.2
> PMDK: false The native code was built without PMDK support.{code}
> The issue seems to be at least two symbols that were removed from ABI in
> OpenSSL 3.x releases:
> * EVP_CIPHER_CTX_block_size (new name: EVP_CIPHER_CTX_get_block_size)
> * EVP_CIPHER_CTX_encrypting (new name: EVP_CIPHER_CTX_is_encrypting)
> The attached patch [^100-hadoop-3.3.4-openssl-3.patch] works around the issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18583. Fix loading of OpenSSL 3.x symbols [hadoop]
steveloughran commented on PR #5256: URL: https://github.com/apache/hadoop/pull/5256#issuecomment-1977458743 > hadoop-yarn-server-nodemanager's container-executor no idea. best to discuss on yarn-dev regarding crypton and checknative, we may to work on that. IMO checknative should look for openssl, at least if we add a -openssl argument. I don't think we have specific openssl issues. In #6425 I had to add error string matching for openssl 1 messages indicating stale https connections (these were surfacing deep in the AWS error stack). For 3.x it'd be good to know that these strings were the same -or update them. Maybe you can create an uber-jira "support openssl 3" one more thing: what is the openssl FIPS story? as for strict fips support we don't just want to talk to fips endpoints, we want to run on hosts where the ssl lib doesn't have the untrusted algorithms at all. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17299. Adding rack failure tolerance when creating a new file [hadoop]
ayushtkn commented on code in PR #6566:
URL: https://github.com/apache/hadoop/pull/6566#discussion_r1511777291
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java:
##
@@ -1817,10 +1839,10 @@ protected LocatedBlock nextBlockOutputStream() throws
IOException {
nodes = lb.getLocations();
nextStorageTypes = lb.getStorageTypes();
nextStorageIDs = lb.getStorageIDs();
-
+ setPipeline(lb);
// Connect to first DataNode in the list.
success = createBlockOutputStream(nodes, nextStorageTypes,
nextStorageIDs,
- 0L, false);
+ 0L, false) || setupPipelineForAppendOrRecovery();
Review Comment:
hmm, to me it looks fine only, I am not sure about any legacy reasons or any
use case which can break due to this. If other folks are ok, should be fine.
@Hexiaoqiao / @tasanuma mind giving a check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19101) Vectored Read into off-heap buffer broken
Steve Loughran created HADOOP-19101:
---
Summary: Vectored Read into off-heap buffer broken
Key: HADOOP-19101
URL: https://issues.apache.org/jira/browse/HADOOP-19101
Project: Hadoop Common
Issue Type: Sub-task
Components: fs, fs/azure
Affects Versions: 3.3.6, 3.4.0
Reporter: Steve Loughran
Assignee: Steve Loughran
{{VectoredReadUtils.readInDirectBuffer()}} always starts off reading at
position zero even when the range is at a different offset. As a result: you
can get incorrect information.
Thanks for this is straightforward: we pass in a FileRange and use its offset
as the starting position.
However, this does mean that all shipping releases 3.3.5-3.4.0 cannot safely
read vectorIO into direct buffers through HDFS, ABFS or Azure. Note that we
have never seen this in production because the parquet and ORC libraries both
read into on-heap storage.
Those libraries needs to be audited to make sure that they never attempt to
read into off-heap DirectBuffers. This is a bit trickier than you would think
because an allocator is passed in. For PARQUET-2171 we will
* only invoke the API on streams which explicitly declare their support for the
API (so fallback in parquet itself)
* not invoke when direct buffer allocation is in use.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18855) VectorIO API tuning
[ https://issues.apache.org/jira/browse/HADOOP-18855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18855: Affects Version/s: 3.4.0 > VectorIO API tuning > --- > > Key: HADOOP-18855 > URL: https://issues.apache.org/jira/browse/HADOOP-18855 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, fs/s3 >Affects Versions: 3.4.0, 3.3.9 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > what do do we need to do to improve the vector iO experience based on > integration and use. > obviously, we cannot change anything incompatibly, but we may find bugs to > fix and other possible enhancements -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18855) VectorIO API tuning
[ https://issues.apache.org/jira/browse/HADOOP-18855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-18855: --- Assignee: Steve Loughran > VectorIO API tuning > --- > > Key: HADOOP-18855 > URL: https://issues.apache.org/jira/browse/HADOOP-18855 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, fs/s3 >Affects Versions: 3.3.9 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > what do do we need to do to improve the vector iO experience based on > integration and use. > obviously, we cannot change anything incompatibly, but we may find bugs to > fix and other possible enhancements -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17383:Datanode current block token should come from active NameNode in HA mode [hadoop]
hadoop-yetus commented on PR #6562:
URL: https://github.com/apache/hadoop/pull/6562#issuecomment-1977312121
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 20s | | trunk passed |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 20s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 17s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 28s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 1m 27s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 27s | | the patch passed |
| +1 :green_heart: | compile | 1m 16s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 13s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6562/4/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 557 unchanged
- 0 fixed = 560 total (was 557) |
| +1 :green_heart: | mvnsite | 1m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 8s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 32s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 49s | | the patch passed |
| +1 :green_heart: | shadedclient | 43m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 250m 18s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6562/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 5s | | The patch does not
generate ASF License warnings. |
| | | 410m 3s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.protocol.TestBlockListAsLongs |
| | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6562/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6562 |
| JIRA Issue | HDFS-17383 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 6239d799761e 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a5e8a0d8c6ee988d8d4eb657fbfe3b44ecf85300 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6562/4/testReport/ |
| Max. process+thread count | 3050 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project
[jira] [Commented] (HADOOP-19098) Vector IO: consistent specified rejection of overlapping ranges
[ https://issues.apache.org/jira/browse/HADOOP-19098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17823287#comment-17823287 ] ASF GitHub Bot commented on HADOOP-19098: - steveloughran commented on PR #6604: URL: https://github.com/apache/hadoop/pull/6604#issuecomment-1977235222 Just going to highlight that the contract test failed *badly* when reading into direct buffers from azure. I think this is a bug in the direct buffer fetching logic -it's always setting 0 as the position for readFully(position...) What does that mean? It means until this patch is in it is *not* safe to read into direct buffers except in the stores which do their own native implementations of the API. which really means "don't use direct buffers as a destination, at all" ``` [ERROR] Tests run: 34, Failures: 13, Errors: 2, Skipped: 0, Time elapsed: 375.803 s <<< FAILURE! - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractVectoredRead [ERROR] testNormalReadAfterVectoredRead[Buffer type : direct](org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractVectoredRead) Time elapsed: 1.607 s <<< FAILURE! java.lang.AssertionError: vecRead with read offset 110: data[0] != DATASET[110] expected:<111> but was:<97> at org.junit.Assert.fail(Assert.java:89) at org.junit.Assert.failNotEquals(Assert.java:835) at org.junit.Assert.assertEquals(Assert.java:647) at org.apache.hadoop.fs.contract.ContractTestUtils.assertDatasetEquals(ContractTestUtils.java:1182) at org.apache.hadoop.fs.contract.ContractTestUtils.validateVectoredReadResult(ContractTestUtils.java:1140) at org.apache.hadoop.fs.contract.AbstractContractVectoredReadTest.testNormalReadAfterVectoredRead(AbstractContractVectoredReadTest.java:347) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) [ERROR] testVectoredReadMultipleRanges[Buffer type : direct](org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractVectoredRead) Time elapsed: 0.391 s <<< FAILURE! java.lang.AssertionError: vecRead with read offset 100: data[0] != DATASET[100] expected:<101> but was:<97> at org.junit.Assert.fail(Assert.java:89) at org.junit.Assert.failNotEquals(Assert.java:835) at org.junit.Assert.assertEquals(Assert.java:647) at org.apache.hadoop.fs.contract.ContractTestUtils.assertDatasetEquals(ContractTestUtils.java:1182) at org.apache.hadoop.fs.contract.ContractTestUtils.validateVectoredReadResult(ContractTestUtils.java:1140) at org.apache.hadoop.fs.contract.AbstractContractVectoredReadTest.testVectoredReadMultipleRanges(AbstractContractVectoredReadTest.java:170) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.eva
Re: [PR] HADOOP-19098 Vector IO: consistent specified rejection of overlapping ranges [hadoop]
steveloughran commented on PR #6604: URL: https://github.com/apache/hadoop/pull/6604#issuecomment-1977235222 Just going to highlight that the contract test failed *badly* when reading into direct buffers from azure. I think this is a bug in the direct buffer fetching logic -it's always setting 0 as the position for readFully(position...) What does that mean? It means until this patch is in it is *not* safe to read into direct buffers except in the stores which do their own native implementations of the API. which really means "don't use direct buffers as a destination, at all" ``` [ERROR] Tests run: 34, Failures: 13, Errors: 2, Skipped: 0, Time elapsed: 375.803 s <<< FAILURE! - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractVectoredRead [ERROR] testNormalReadAfterVectoredRead[Buffer type : direct](org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractVectoredRead) Time elapsed: 1.607 s <<< FAILURE! java.lang.AssertionError: vecRead with read offset 110: data[0] != DATASET[110] expected:<111> but was:<97> at org.junit.Assert.fail(Assert.java:89) at org.junit.Assert.failNotEquals(Assert.java:835) at org.junit.Assert.assertEquals(Assert.java:647) at org.apache.hadoop.fs.contract.ContractTestUtils.assertDatasetEquals(ContractTestUtils.java:1182) at org.apache.hadoop.fs.contract.ContractTestUtils.validateVectoredReadResult(ContractTestUtils.java:1140) at org.apache.hadoop.fs.contract.AbstractContractVectoredReadTest.testNormalReadAfterVectoredRead(AbstractContractVectoredReadTest.java:347) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) [ERROR] testVectoredReadMultipleRanges[Buffer type : direct](org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractVectoredRead) Time elapsed: 0.391 s <<< FAILURE! java.lang.AssertionError: vecRead with read offset 100: data[0] != DATASET[100] expected:<101> but was:<97> at org.junit.Assert.fail(Assert.java:89) at org.junit.Assert.failNotEquals(Assert.java:835) at org.junit.Assert.assertEquals(Assert.java:647) at org.apache.hadoop.fs.contract.ContractTestUtils.assertDatasetEquals(ContractTestUtils.java:1182) at org.apache.hadoop.fs.contract.ContractTestUtils.validateVectoredReadResult(ContractTestUtils.java:1140) at org.apache.hadoop.fs.contract.AbstractContractVectoredReadTest.testVectoredReadMultipleRanges(AbstractContractVectoredReadTest.java:170) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statem
Re: [PR] HDFS-17408:Reduce the number of quota calculations in FSDirRenameOp [hadoop]
hadoop-yetus commented on PR #6608:
URL: https://github.com/apache/hadoop/pull/6608#issuecomment-1977212481
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 6m 42s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 18s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 42s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 38s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 4s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 45s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 39s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 38s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 38s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 30s | |
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 46 unchanged - 1
fixed = 46 total (was 47) |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 0s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 45s | | the patch passed |
| -1 :x: | shadedclient | 20m 21s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 203m 30s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6608/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 297m 35s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.cli.TestAclCLI |
| | hadoop.hdfs.TestDistributedFileSystem |
| |
hadoop.hdfs.server.namenode.snapshot.TestFSImageWithOrderedSnapshotDeletion |
| | hadoop.hdfs.web.TestWebHDFS |
| | hadoop.hdfs.server.mover.TestStorageMover |
| | hadoop.hdfs.protocol.TestBlockListAsLongs |
| | hadoop.hdfs.TestQuota |
| | hadoop.hdfs.TestRenameWhileOpen |
| | hadoop.fs.TestWebHdfsFileContextMainOperations |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestProvidedImpl |
| | hadoop.hdfs.TestEncryptionZones |
| | hadoop.cli.TestHDFSCLI |
| | hadoop.hdfs.TestFileCreation |
| | hadoop.hdfs.server.namenode.snapshot.TestRandomOpsWithSnapshots |
| |
hadoop.hdfs.server.namenode.snapshot.TestFsShellMoveToTrashWithSnapshots |
| | hadoop.hdfs.server.namenode.TestNameNodeXAttr |
| | hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy |
| | hadoop.hdfs.server.namenode.TestFsLimits |
| | hadoop.fs.viewfs.TestViewFsHdfs |
| | hadoop.hdfs.server.namenode.TestProtectedDirectories |
| | hadoop.hdfs.TestTrashWithSecureEncryptionZones |
| | hadoop.hdfs.TestDatanodeLayoutUpgrade |
| | hadoop.fs.viewfs.TestViewFSOverloadSchemeWithMountTableConfigInHDFS |
| | hadoop.fs.viewfs.TestViewFileSystemLinkMergeSlash |
| | hadoop.hdfs.server.namenode.TestHDFS
Re: [PR] HDFS-17398. [FGL] Implement the FGL lock for FSNLockManager [hadoop]
hadoop-yetus commented on PR #6599:
URL: https://github.com/apache/hadoop/pull/6599#issuecomment-1977198415
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 16m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ HDFS-17384 Compile Tests _ |
| +1 :green_heart: | mvninstall | 48m 27s | | HDFS-17384 passed |
| +1 :green_heart: | compile | 1m 25s | | HDFS-17384 passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 15s | | HDFS-17384 passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 12s | | HDFS-17384 passed |
| +1 :green_heart: | mvnsite | 1m 22s | | HDFS-17384 passed |
| +1 :green_heart: | javadoc | 1m 10s | | HDFS-17384 passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 35s | | HDFS-17384 passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 17s | | HDFS-17384 passed |
| +1 :green_heart: | shadedclient | 40m 0s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 10s | | the patch passed |
| +1 :green_heart: | compile | 1m 16s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 16s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 3s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6599/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 0 unchanged -
0 fixed = 2 total (was 0) |
| +1 :green_heart: | mvnsite | 1m 14s | | the patch passed |
| -1 :x: | javadoc | 0m 56s |
[/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6599/2/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| +1 :green_heart: | javadoc | 1m 31s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 47s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 259m 23s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6599/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 428m 47s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.tools.TestDFSAdmin |
| | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| | hadoop.hdfs.protocol.TestBlockListAsLongs |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6599/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6599 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b57cdf3721f1 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | HDFS-17384 / ce0e01f1f5988faf36ef19cc7c5838038f2d530c |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-
[jira] [Comment Edited] (HADOOP-19091) Add support for Tez to MagicS3GuardCommitter
[
https://issues.apache.org/jira/browse/HADOOP-19091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17823282#comment-17823282
]
Venkatasubrahmanian Narayanan edited comment on HADOOP-19091 at 3/4/24 6:23 PM:
[~srahman] While that's true, the problem is that the jobAttemptPath itself is
different between task and AM. If you look at my previous message, you'll see
the difference is in the "directory" name of the jobAttemptPath:
(spaces inserted below because Jira doesn't like double underscore)
Task: s3a://hive-east-1-bucket/emblembasic/ _ _ magic/ __ __
magic/job-job_17089738741890_0073/
vs
AM: s3a://hive-east-1-bucket/emblembasic/ _ _ magic/ _ _
magic/job-job_1708973874189_0073/
The extra 0 in the JobID causes them to look at different "directories", and
hence it doesn't find it.
There isn't a stacktrace per se - the commitJob op just doesn't find any
pending data to commit, so it just goes to the cleanup() code(where since my
test are with Hadoop 3.3.3, it just looks under __magic and finds everything to
be deleted).
was (Author: vnarayanan7):
[~srahman] While that's true, the problem is that the jobAttemptPath itself is
different between task and AM. If you look at my previous message, you'll see
the difference is in the "directory" name of the jobAttemptPath:
Task:
s3a://hive-east-1-bucket/emblembasic/__magic/__magic/job-job_17089738741890_0073/
vs
AM:
s3a://hive-east-1-bucket/emblembasic/_{_}_{_}magic/__magic/job-job_1708973874189_0073/
The extra 0 in the JobID causes them to look at different "directories", and
hence it doesn't find it.
There isn't a stacktrace per se - the commitJob op just doesn't find any
pending data to commit, so it just goes to the cleanup() code(where since my
test are with Hadoop 3.3.3, it just looks under __magic and finds everything to
be deleted).
> Add support for Tez to MagicS3GuardCommitter
>
>
> Key: HADOOP-19091
> URL: https://issues.apache.org/jira/browse/HADOOP-19091
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Affects Versions: 3.3.6
> Environment: Pig 17/Hive 3.1.3 with Hadoop 3.3.3 on AWS EMR 6-12.0
>Reporter: Venkatasubrahmanian Narayanan
>Assignee: Venkatasubrahmanian Narayanan
>Priority: Major
> Attachments: 0001-AWS-Hive-Changes.patch,
> 0002-HIVE-27698-Backport-of-HIVE-22398-Remove-legacy-code.patch,
> HADOOP-19091-HIVE-WIP.patch
>
>
> The MagicS3GuardCommitter assumes that the JobID of the task is the same as
> that of the job's application master when writing/reading the .pendingset
> file. This assumption is not valid when running with Tez, which creates
> slightly different JobIDs for tasks and the application master.
>
> While the MagicS3GuardCommitter is intended only for MRv2, it mostly works
> fine with an MRv1 wrapper with Hive/Pig (with some minor changes to Hive) run
> in MR mode. This issue only crops up when running queries with the Tez
> execution engine. I can upload a patch to Hive 3.1 to reproduce this error on
> EMR if needed.
>
> Fixing this will probably require work from both Tez and Hadoop, wanted to
> start a discussion here so we can figure out how exactly we go about this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19091) Add support for Tez to MagicS3GuardCommitter
[
https://issues.apache.org/jira/browse/HADOOP-19091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17823282#comment-17823282
]
Venkatasubrahmanian Narayanan commented on HADOOP-19091:
[~srahman] While that's true, the problem is that the jobAttemptPath itself is
different between task and AM. If you look at my previous message, you'll see
the difference is in the "directory" name of the jobAttemptPath:
Task:
s3a://hive-east-1-bucket/emblembasic/__{_}magic/{_}__magic/job-job_17089738741890_0073/
vs
AM:
s3a://hive-east-1-bucket/emblembasic/__{_}magic/__{_}{_}magic/job-job_1708973874189_0073/{_}
The extra 0 in the JobID causes them to look at different "directories", and
hence it doesn't find it.
There isn't a stacktrace per se - the commitJob op just doesn't find any
pending data to commit, so it just goes to the cleanup() code(where since my
test are with Hadoop 3.3.3, it just looks under __magic and finds everything to
be deleted).
> Add support for Tez to MagicS3GuardCommitter
>
>
> Key: HADOOP-19091
> URL: https://issues.apache.org/jira/browse/HADOOP-19091
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Affects Versions: 3.3.6
> Environment: Pig 17/Hive 3.1.3 with Hadoop 3.3.3 on AWS EMR 6-12.0
>Reporter: Venkatasubrahmanian Narayanan
>Assignee: Venkatasubrahmanian Narayanan
>Priority: Major
> Attachments: 0001-AWS-Hive-Changes.patch,
> 0002-HIVE-27698-Backport-of-HIVE-22398-Remove-legacy-code.patch,
> HADOOP-19091-HIVE-WIP.patch
>
>
> The MagicS3GuardCommitter assumes that the JobID of the task is the same as
> that of the job's application master when writing/reading the .pendingset
> file. This assumption is not valid when running with Tez, which creates
> slightly different JobIDs for tasks and the application master.
>
> While the MagicS3GuardCommitter is intended only for MRv2, it mostly works
> fine with an MRv1 wrapper with Hive/Pig (with some minor changes to Hive) run
> in MR mode. This issue only crops up when running queries with the Tez
> execution engine. I can upload a patch to Hive 3.1 to reproduce this error on
> EMR if needed.
>
> Fixing this will probably require work from both Tez and Hadoop, wanted to
> start a discussion here so we can figure out how exactly we go about this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-19091) Add support for Tez to MagicS3GuardCommitter
[
https://issues.apache.org/jira/browse/HADOOP-19091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17823282#comment-17823282
]
Venkatasubrahmanian Narayanan edited comment on HADOOP-19091 at 3/4/24 6:22 PM:
[~srahman] While that's true, the problem is that the jobAttemptPath itself is
different between task and AM. If you look at my previous message, you'll see
the difference is in the "directory" name of the jobAttemptPath:
Task:
s3a://hive-east-1-bucket/emblembasic/__magic/__magic/job-job_17089738741890_0073/
vs
AM:
s3a://hive-east-1-bucket/emblembasic/_{_}_{_}magic/__magic/job-job_1708973874189_0073/
The extra 0 in the JobID causes them to look at different "directories", and
hence it doesn't find it.
There isn't a stacktrace per se - the commitJob op just doesn't find any
pending data to commit, so it just goes to the cleanup() code(where since my
test are with Hadoop 3.3.3, it just looks under __magic and finds everything to
be deleted).
was (Author: vnarayanan7):
[~srahman] While that's true, the problem is that the jobAttemptPath itself is
different between task and AM. If you look at my previous message, you'll see
the difference is in the "directory" name of the jobAttemptPath:
Task:
s3a://hive-east-1-bucket/emblembasic/__{_}magic/{_}__magic/job-job_17089738741890_0073/
vs
AM:
s3a://hive-east-1-bucket/emblembasic/__{_}magic/__{_}{_}magic/job-job_1708973874189_0073/{_}
The extra 0 in the JobID causes them to look at different "directories", and
hence it doesn't find it.
There isn't a stacktrace per se - the commitJob op just doesn't find any
pending data to commit, so it just goes to the cleanup() code(where since my
test are with Hadoop 3.3.3, it just looks under __magic and finds everything to
be deleted).
> Add support for Tez to MagicS3GuardCommitter
>
>
> Key: HADOOP-19091
> URL: https://issues.apache.org/jira/browse/HADOOP-19091
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Affects Versions: 3.3.6
> Environment: Pig 17/Hive 3.1.3 with Hadoop 3.3.3 on AWS EMR 6-12.0
>Reporter: Venkatasubrahmanian Narayanan
>Assignee: Venkatasubrahmanian Narayanan
>Priority: Major
> Attachments: 0001-AWS-Hive-Changes.patch,
> 0002-HIVE-27698-Backport-of-HIVE-22398-Remove-legacy-code.patch,
> HADOOP-19091-HIVE-WIP.patch
>
>
> The MagicS3GuardCommitter assumes that the JobID of the task is the same as
> that of the job's application master when writing/reading the .pendingset
> file. This assumption is not valid when running with Tez, which creates
> slightly different JobIDs for tasks and the application master.
>
> While the MagicS3GuardCommitter is intended only for MRv2, it mostly works
> fine with an MRv1 wrapper with Hive/Pig (with some minor changes to Hive) run
> in MR mode. This issue only crops up when running queries with the Tez
> execution engine. I can upload a patch to Hive 3.1 to reproduce this error on
> EMR if needed.
>
> Fixing this will probably require work from both Tez and Hadoop, wanted to
> start a discussion here so we can figure out how exactly we go about this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] HDFS-17404: Add NN Socket Address to log when processing command from active NN [hadoop]
Jtdellaringa opened a new pull request, #6609: URL: https://github.com/apache/hadoop/pull/6609 ### Description of PR Add NN Socket Address to log when processing command from active NN ### How was this patch tested? n/a ### For code changes: - [ x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17404. Add Namenode info to log message when setting block keys [hadoop]
Jtdellaringa commented on code in PR #6598:
URL: https://github.com/apache/hadoop/pull/6598#discussion_r1511568416
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenSecretManager.java:
##
@@ -226,7 +228,7 @@ public synchronized void addKeys(ExportedBlockKeys
exportedKeys)
if (isMaster || exportedKeys == null) {
return;
}
-LOG.info("Setting block keys. BlockPool = {} .", blockPoolId);
+LOG.info("Setting block keys. BlockPool = {} . NamenodeIndex = {} .",
blockPoolId, nnIndex);
Review Comment:
Decided not to pass BPServiceActor due to it being a private class. Namenode
address will instead be logged when dn received command from standby or active
nn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17299. Adding rack failure tolerance when creating a new file [hadoop]
shahrs87 commented on code in PR #6566:
URL: https://github.com/apache/hadoop/pull/6566#discussion_r1511445056
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java:
##
@@ -1817,10 +1839,10 @@ protected LocatedBlock nextBlockOutputStream() throws
IOException {
nodes = lb.getLocations();
nextStorageTypes = lb.getStorageTypes();
nextStorageIDs = lb.getStorageIDs();
-
+ setPipeline(lb);
// Connect to first DataNode in the list.
success = createBlockOutputStream(nodes, nextStorageTypes,
nextStorageIDs,
- 0L, false);
+ 0L, false) || setupPipelineForAppendOrRecovery();
Review Comment:
> If I catch it right, it is a behaviour change, right?
I agree this is a behavior change but the current behavior is buggy.
`dtpReplaceDatanodeOnFailureReplication` configuration is NOT honored during
the `PIPELINE_SETUP_CREATE` phase. Thats why this jira was created.
After this patch, we will handle datanode failure during
`PIPELINE_SETUP_CREATE` in the same way as `DATA_STREAMING` phase and the
application (in this case hbase) will experience consistent behavior using hdfs
client.
Cc @ritegarg @ayushtkn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] Hadoop 18325: ABFS: Add correlated metric support for ABFS operations [hadoop]
hadoop-yetus commented on PR #6314:
URL: https://github.com/apache/hadoop/pull/6314#issuecomment-1976952564
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 28s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 8 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 6s | | trunk passed |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 38s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 32m 59s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 33m 20s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 20s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6314/8/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 11 new + 10 unchanged - 0
fixed = 21 total (was 10) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 32m 46s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 14s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 143m 16s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6314/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux 3e7deb629c50 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0395e945c119294793ed90603822231ccb73478f |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6314/8/testReport/ |
| Max. process+thread count | 714 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job
Re: [PR] HDFS-17391:Adjust the checkpoint io buffer size to the chunk size [hadoop]
hadoop-yetus commented on PR #6594:
URL: https://github.com/apache/hadoop/pull/6594#issuecomment-1976950545
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 7m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 19s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 36s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 2s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 42s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 20s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 38s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 30s | |
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 4 unchanged - 4
fixed = 4 total (was 8) |
| +1 :green_heart: | mvnsite | 0m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 42s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 14s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 121m 45s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6594/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +0 :ok: | asflicense | 0m 27s | | ASF License check generated no
output? |
| | | 215m 50s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.blockmanagement.TestBlockManager |
| | hadoop.hdfs.server.namenode.TestCacheDirectives |
| | hadoop.hdfs.server.namenode.TestSecondaryWebUi |
| | hadoop.hdfs.tools.TestDFSAdmin |
| | hadoop.hdfs.server.namenode.TestAuditLogger |
| |
hadoop.hdfs.server.namenode.snapshot.TestFsShellMoveToTrashWithSnapshots |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.server.namenode.TestListOpenFiles |
| | hadoop.hdfs.server.namenode.TestDefaultBlockPlacementPolicy |
| | hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand |
| | hadoop.hdfs.server.namenode.TestXAttrConfigFlag |
| | hadoop.hdfs.server.namenode.TestAddBlock |
| | hadoop.hdfs.server.namenode.TestNetworkTopologyServlet |
| | hadoop.hdfs.server.namenode.TestUpgradeDomainBlockPlacementPolicy |
| | hadoop.hdfs.protocol.TestBlockListAsLongs |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6594/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6594 |
| JIRA Issue | HDFS-17391 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell d
Re: [PR] Hadoop 18325: ABFS: Add correlated metric support for ABFS operations [hadoop]
hadoop-yetus commented on PR #6314:
URL: https://github.com/apache/hadoop/pull/6314#issuecomment-1976675970
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 23s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 8 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 38s | | trunk passed |
| +1 :green_heart: | compile | 0m 21s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 20s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 18s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 23s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 20s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 15s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 27s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 18s | | the patch passed |
| +1 :green_heart: | compile | 0m 17s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 17s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 11s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6314/7/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 11 new + 10 unchanged - 0
fixed = 21 total (was 10) |
| +1 :green_heart: | mvnsite | 0m 17s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 15s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 15s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 38s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 54s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 51s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 24s | | The patch does not
generate ASF License warnings. |
| | | 85m 11s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6314/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux 83936f6c7696 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 901f6706945a2e38b2d58fc8edeb529164f9de02 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6314/7/testReport/ |
| Max. process+thread count | 716 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job
Re: [PR] Hadoop 18325: ABFS: Add correlated metric support for ABFS operations [hadoop]
hadoop-yetus commented on PR #6314:
URL: https://github.com/apache/hadoop/pull/6314#issuecomment-1976639659
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 8 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 3s | | trunk passed |
| +1 :green_heart: | compile | 0m 26s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 21s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 19s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 20s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 40s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 57s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 10s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 18s | | the patch passed |
| +1 :green_heart: | compile | 0m 17s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 17s | | the patch passed |
| +1 :green_heart: | compile | 0m 15s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 10s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6314/6/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 6 new + 10 unchanged - 0
fixed = 16 total (was 10) |
| +1 :green_heart: | mvnsite | 0m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 15s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 16s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 40s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 7s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 46s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 23s | | The patch does not
generate ASF License warnings. |
| | | 85m 29s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6314/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6314 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile
javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle
markdownlint |
| uname | Linux 0450eb8c5390 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f45c3db155233615f13e6fb5769d8256759a74fe |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6314/6/testReport/ |
| Max. process+thread count | 684 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
[PR] HDFS-17408:Reduce the number of quota calculations in FSDirRenameOp [hadoop]
ThinkerLei opened a new pull request, #6608: URL: https://github.com/apache/hadoop/pull/6608 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19097) core-default fs.s3a.connection.establish.timeout value too low -warning always printed
[
https://issues.apache.org/jira/browse/HADOOP-19097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17823153#comment-17823153
]
ASF GitHub Bot commented on HADOOP-19097:
-
steveloughran commented on PR #6601:
URL: https://github.com/apache/hadoop/pull/6601#issuecomment-1976473456
@virajjasani I did a scan. Now, the clever thing would be to have a test
suite which compared values so there is never a regression. e.g
assertDurationEqual(conf, CONNECTION_TIMEOUT, CONNECTION_TIMEOUT_DEFAULT)
> core-default fs.s3a.connection.establish.timeout value too low -warning
> always printed
> --
>
> Key: HADOOP-19097
> URL: https://issues.apache.org/jira/browse/HADOOP-19097
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Labels: pull-request-available
>
> caused by HADOOP-18915.
> in core-default we set the value of fs.s3a.connection.establish.timeout to 5s
> {code}
>
> fs.s3a.connection.establish.timeout
> 5s
>
> {code}
> but there is a minimum of 15s, so this prints a warning
> {code}
> 2024-02-29 10:39:27,369 WARN impl.ConfigurationHelper: Option
> fs.s3a.connection.establish.timeout is too low (5,000 ms). Setting to 15,000
> ms instead
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19097. S3A: Set fs.s3a.connection.establish.timeout to 30s [hadoop]
steveloughran commented on PR #6601: URL: https://github.com/apache/hadoop/pull/6601#issuecomment-1976473456 @virajjasani I did a scan. Now, the clever thing would be to have a test suite which compared values so there is never a regression. e.g assertDurationEqual(conf, CONNECTION_TIMEOUT, CONNECTION_TIMEOUT_DEFAULT) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17387. [FGL] Abstract the configuration locking mode [hadoop]
ferhui commented on PR #6572: URL: https://github.com/apache/hadoop/pull/6572#issuecomment-1976195600 Thanks. Merged -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17387. [FGL] Abstract the configuration locking mode [hadoop]
ferhui merged PR #6572: URL: https://github.com/apache/hadoop/pull/6572 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17387. [FGL] Abstract the configuration locking mode [hadoop]
ZanderXu commented on PR #6572: URL: https://github.com/apache/hadoop/pull/6572#issuecomment-1976061312 @ferhui The failed UTs are not introduced by this PR. I will submit a new PR to fix them and contribute them to the trunk branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17387. [FGL] Abstract the configuration locking mode [hadoop]
hadoop-yetus commented on PR #6572:
URL: https://github.com/apache/hadoop/pull/6572#issuecomment-1976013604
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 7m 2s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ HDFS-17384 Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 52s | | HDFS-17384 passed |
| +1 :green_heart: | compile | 0m 46s | | HDFS-17384 passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 39s | | HDFS-17384 passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 40s | | HDFS-17384 passed |
| +1 :green_heart: | mvnsite | 0m 45s | | HDFS-17384 passed |
| +1 :green_heart: | javadoc | 0m 42s | | HDFS-17384 passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 3s | | HDFS-17384 passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 46s | | HDFS-17384 passed |
| +1 :green_heart: | shadedclient | 20m 35s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 38s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 30s | |
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 310 unchanged - 10
fixed = 310 total (was 320) |
| +1 :green_heart: | mvnsite | 0m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 45s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 36s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 201m 41s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6572/10/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not
generate ASF License warnings. |
| | | 298m 43s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.protocol.TestBlockListAsLongs |
| | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| | hadoop.hdfs.tools.TestDFSAdmin |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6572/10/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6572 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 084b55dedc03 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | HDFS-17384 / 50207fa1b5c523991566fbcf9d37169d9ee3ecf8 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/had
[jira] [Commented] (HADOOP-19085) Compatibility Benchmark over HCFS Implementations
[
https://issues.apache.org/jira/browse/HADOOP-19085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17823076#comment-17823076
]
Kai Zheng commented on HADOOP-19085:
Hi [~han.liu] The work looks great overall. Some comments follow and would you
help check? Thanks!
1. About the document HdfsCompatBenchIssue.md, maybe we could reshape it as the
user doc like we have for other hadoop-tools modules.
2. For most classes in the package org/apache/hadoop/fs/compat, we could have a
sub package like 'common' for them, so we could leave the main package much
cleaner.
3. The tool (HdfsCompatibility.java) could be 'HdfsCompatTool'?
4. HdfsCompatFileSystemImpl could be 'HdfsCompatBasics' and moved to stay along
with the function cases together. Actually we could put all the HCFS API cases
together directly under the 'cases' package.
5. We could rename 'modification.t' to 'write.t', and have another file like
'directory.t' for dir related operations.
> Compatibility Benchmark over HCFS Implementations
> -
>
> Key: HADOOP-19085
> URL: https://issues.apache.org/jira/browse/HADOOP-19085
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Han Liu
>Assignee: Han Liu
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS Compatibility Benchmark Design.pdf
>
>
> {*}Background：{*}Hadoop-Compatible File System (HCFS) is a core conception in
> big data storage ecosystem, providing unified interfaces and generally clear
> semantics, and has become the de-factor standard for industry storage systems
> to follow and conform with. There have been a series of HCFS implementations
> in Hadoop, such as S3AFileSystem for Amazon's S3 Object Store, WASB for
> Microsoft's Azure Blob Storage and OSS connector for Alibaba Cloud Object
> Storage, and more from storage service's providers on their own.
> {*}Problems：{*}However, as indicated by introduction.md, there is no formal
> suite to do compatibility assessment of a file system for all such HCFS
> implementations. Thus, whether the functionality is well accomplished and
> meets the core compatible expectations mainly relies on service provider's
> own report. Meanwhile, Hadoop is also developing and new features are
> continuously contributing to HCFS interfaces for existing implementations to
> follow and update, in which case, Hadoop also needs a tool to quickly assess
> if these features are supported or not for a specific HCFS implementation.
> Besides, the known hadoop command line tool or hdfs shell is used to directly
> interact with a HCFS storage system, where most commands correspond to
> specific HCFS interfaces and work well. Still, there are cases that are
> complicated and may not work, like expunge command. To check such commands
> for an HCFS, we also need an approach to figure them out.
> {*}Proposal：{*}Accordingly, we propose to define a formal HCFS compatibility
> benchmark and provide corresponding tool to do the compatibility assessment
> for an HCFS storage system. The benchmark and tool should consider both HCFS
> interfaces and hdfs shell commands. Different scenarios require different
> kinds of compatibilities. For such consideration, we could define different
> suites in the benchmark.
> *Benefits:* We intend the benchmark and tool to be useful for both storage
> providers and storage users. For end users, it can be used to evalute the
> compatibility level and determine if the storage system in question is
> suitable for the required scenarios. For storage providers, it helps to
> quickly generate an objective and reliable report about core functioins of
> the storage service. As an instance, if the HCFS got a 100% on a suite named
> 'tpcds', it is demonstrated that all functions needed by a tpcds program have
> been well achieved. It is also a guide indicating how storage service
> abilities can map to HCFS interfaces, such as storage class on S3.
> Any thoughts? Comments and feedback are mostly welcomed. Thanks in advance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] Hadoop 18325: ABFS: Add correlated metric support for ABFS operations [hadoop]
saxenapranav commented on code in PR #6314:
URL: https://github.com/apache/hadoop/pull/6314#discussion_r1510743018
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##
@@ -1411,6 +1448,97 @@ protected AccessTokenProvider getTokenProvider() {
return tokenProvider;
}
+ public AzureBlobFileSystem getMetricFilesystem() throws IOException {
+if (metricFs == null) {
+ try {
+Configuration metricConfig = abfsConfiguration.getRawConfiguration();
+String metricAccountKey =
metricConfig.get(FS_AZURE_METRIC_ACCOUNT_KEY);
+final String abfsMetricUrl = metricConfig.get(FS_AZURE_METRIC_URI);
+if (abfsMetricUrl == null) {
+ return null;
+}
+metricConfig.set(FS_AZURE_ACCOUNT_KEY_PROPERTY_NAME, metricAccountKey);
+metricConfig.set(AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION,
"false");
+URI metricUri;
+metricUri = new URI(FileSystemUriSchemes.ABFS_SCHEME, abfsMetricUrl,
null, null, null);
+metricFs = (AzureBlobFileSystem) FileSystem.newInstance(metricUri,
metricConfig);
Review Comment:
Yes, because we donot want any functionality of fileSystem, but just want to
trigger a server call to metric uri and would remove circular dependency. In
the customized method, it would take metricUri, create the restOperation and
then execute it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] Hadoop 18325: ABFS: Add correlated metric support for ABFS operations [hadoop]
saxenapranav commented on code in PR #6314:
URL: https://github.com/apache/hadoop/pull/6314#discussion_r1510743018
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##
@@ -1411,6 +1448,97 @@ protected AccessTokenProvider getTokenProvider() {
return tokenProvider;
}
+ public AzureBlobFileSystem getMetricFilesystem() throws IOException {
+if (metricFs == null) {
+ try {
+Configuration metricConfig = abfsConfiguration.getRawConfiguration();
+String metricAccountKey =
metricConfig.get(FS_AZURE_METRIC_ACCOUNT_KEY);
+final String abfsMetricUrl = metricConfig.get(FS_AZURE_METRIC_URI);
+if (abfsMetricUrl == null) {
+ return null;
+}
+metricConfig.set(FS_AZURE_ACCOUNT_KEY_PROPERTY_NAME, metricAccountKey);
+metricConfig.set(AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION,
"false");
+URI metricUri;
+metricUri = new URI(FileSystemUriSchemes.ABFS_SCHEME, abfsMetricUrl,
null, null, null);
+metricFs = (AzureBlobFileSystem) FileSystem.newInstance(metricUri,
metricConfig);
Review Comment:
Yes, because we donot want any functionality of fileSystem, but just want to
trigger a server call to metric uri and would remove circular dependency. In
the customized method, it take metricUri, create the restOperation and then
execute it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] Hadoop 18325: ABFS: Add correlated metric support for ABFS operations [hadoop]
anmolanmol1234 commented on code in PR #6314:
URL: https://github.com/apache/hadoop/pull/6314#discussion_r1510739331
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##
@@ -1411,6 +1448,97 @@ protected AccessTokenProvider getTokenProvider() {
return tokenProvider;
}
+ public AzureBlobFileSystem getMetricFilesystem() throws IOException {
+if (metricFs == null) {
+ try {
+Configuration metricConfig = abfsConfiguration.getRawConfiguration();
+String metricAccountKey =
metricConfig.get(FS_AZURE_METRIC_ACCOUNT_KEY);
+final String abfsMetricUrl = metricConfig.get(FS_AZURE_METRIC_URI);
+if (abfsMetricUrl == null) {
+ return null;
+}
+metricConfig.set(FS_AZURE_ACCOUNT_KEY_PROPERTY_NAME, metricAccountKey);
+metricConfig.set(AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION,
"false");
+URI metricUri;
+metricUri = new URI(FileSystemUriSchemes.ABFS_SCHEME, abfsMetricUrl,
null, null, null);
+metricFs = (AzureBlobFileSystem) FileSystem.newInstance(metricUri,
metricConfig);
Review Comment:
So is the suggestion that we don't create a new filesystem instance, just
use the URL for execution ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
