Re: [PR] Hadoop 18325: ABFS: Add correlated metric support for ABFS operations [hadoop]
anujmodi2021 commented on code in PR #6314:
URL: https://github.com/apache/hadoop/pull/6314#discussion_r1527924421
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java:
##
@@ -700,6 +698,18 @@ public synchronized void close() throws IOException {
if (isClosed) {
return;
}
+if (abfsStore.getClient().isMetricCollectionEnabled()) {
+ TracingContext tracingMetricContext = new TracingContext(
+ clientCorrelationId,
+ fileSystemId, FSOperationType.GET_ATTR, true,
Review Comment:
Here for metric tracing context we are using fileSystemId only, where as the
metric tracing context created in ABfsClient, we are suing hostname...
Any reason for this difference?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17299. Adding rack failure tolerance when creating a new file [hadoop]
hadoop-yetus commented on PR #6614:
URL: https://github.com/apache/hadoop/pull/6614#issuecomment-2003013056
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ branch-2.10 Compile Tests _ |
| +0 :ok: | mvndep | 2m 27s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 13m 5s | | branch-2.10 passed |
| +1 :green_heart: | compile | 2m 11s | | branch-2.10 passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | compile | 1m 48s | | branch-2.10 passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| +1 :green_heart: | checkstyle | 0m 46s | | branch-2.10 passed |
| +1 :green_heart: | mvnsite | 1m 47s | | branch-2.10 passed |
| +1 :green_heart: | javadoc | 1m 56s | | branch-2.10 passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javadoc | 1m 20s | | branch-2.10 passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| -1 :x: | spotbugs | 2m 44s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6614/9/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html)
| hadoop-hdfs-project/hadoop-hdfs in branch-2.10 has 1 extant spotbugs
warnings. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 33s | | the patch passed |
| +1 :green_heart: | compile | 2m 7s | | the patch passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javac | 2m 7s | | the patch passed |
| +1 :green_heart: | compile | 1m 44s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| +1 :green_heart: | javac | 1m 44s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 40s | | hadoop-hdfs-project: The
patch generated 0 new + 283 unchanged - 2 fixed = 283 total (was 285) |
| +1 :green_heart: | mvnsite | 1m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 46s | | the patch passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javadoc | 1m 12s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| +1 :green_heart: | spotbugs | 4m 44s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 28s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 97m 4s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6614/9/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 153m 25s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| | hadoop.hdfs.TestFileLengthOnClusterRestart |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6614/9/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6614 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 4fef73a925b4 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-2.10 / 74af92713123574b90dac1bc60fb86bf31cae1c7 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/zulu-7-amd64:Azul Systems,
Inc.-1.7.0_262-b10 /usr/lib/jvm/java
[jira] [Commented] (HADOOP-14837) Handle S3A "glacier" data
[ https://issues.apache.org/jira/browse/HADOOP-14837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827847#comment-17827847 ] ASF GitHub Bot commented on HADOOP-14837: - bpahuja commented on PR #6407: URL: https://github.com/apache/hadoop/pull/6407#issuecomment-2002993100 Hello @steveloughran, Just a gentle reminder to review the PR :) , Do take a look whenever you get some time Thanks > Handle S3A "glacier" data > - > > Key: HADOOP-14837 > URL: https://issues.apache.org/jira/browse/HADOOP-14837 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Bhavay Pahuja >Priority: Minor > Labels: pull-request-available > > SPARK-21797 covers how if you have AWS S3 set to copy some files to glacier, > they appear in the listing but GETs fail, and so does everything else > We should think about how best to handle this. > # report better > # if listings can identify files which are glaciated then maybe we could have > an option to filter them out > # test & see what happens -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-14837 : Support Read Restored Glacier Objects [hadoop]
bpahuja commented on PR #6407: URL: https://github.com/apache/hadoop/pull/6407#issuecomment-2002993100 Hello @steveloughran, Just a gentle reminder to review the PR :) , Do take a look whenever you get some time Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] WIP: ApacheHttpClient adaptation in ABFS. [hadoop]
saxenapranav opened a new pull request, #6633: URL: https://github.com/apache/hadoop/pull/6633 WIP Draft PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19088) upgrade to jersey-json 1.22.0
[ https://issues.apache.org/jira/browse/HADOOP-19088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827846#comment-17827846 ] ASF GitHub Bot commented on HADOOP-19088: - dineshchitlangia merged PR #6627: URL: https://github.com/apache/hadoop/pull/6627 > upgrade to jersey-json 1.22.0 > - > > Key: HADOOP-19088 > URL: https://issues.apache.org/jira/browse/HADOOP-19088 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.3.6 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > Tidies up support for Jettison and Jackson versions used by Hadoop -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19088. Use jersey-json 1.22.0 (#6585) [hadoop]
dineshchitlangia merged PR #6627: URL: https://github.com/apache/hadoop/pull/6627 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17299. Adding rack failure tolerance when creating a new file [hadoop]
hadoop-yetus commented on PR #6614:
URL: https://github.com/apache/hadoop/pull/6614#issuecomment-2002971355
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 22s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ branch-2.10 Compile Tests _ |
| +0 :ok: | mvndep | 2m 24s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 10m 26s | | branch-2.10 passed |
| +1 :green_heart: | compile | 1m 30s | | branch-2.10 passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | compile | 1m 12s | | branch-2.10 passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| +1 :green_heart: | checkstyle | 0m 29s | | branch-2.10 passed |
| +1 :green_heart: | mvnsite | 1m 16s | | branch-2.10 passed |
| +1 :green_heart: | javadoc | 1m 17s | | branch-2.10 passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javadoc | 0m 56s | | branch-2.10 passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| -1 :x: | spotbugs | 1m 43s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6614/8/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html)
| hadoop-hdfs-project/hadoop-hdfs in branch-2.10 has 1 extant spotbugs
warnings. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 18s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 3s | | the patch passed |
| +1 :green_heart: | compile | 1m 25s | | the patch passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javac | 1m 25s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 24s | | hadoop-hdfs-project: The
patch generated 0 new + 283 unchanged - 2 fixed = 283 total (was 285) |
| +1 :green_heart: | mvnsite | 1m 6s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 11s | | the patch passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| +1 :green_heart: | spotbugs | 2m 49s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 13s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 74m 5s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6614/8/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 25s | | The patch does not
generate ASF License warnings. |
| | | 113m 47s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.web.TestWebHDFS |
| | hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.TestDFSInotifyEventInputStream |
| | hadoop.hdfs.TestLeaseRecovery2 |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6614/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6614 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 20973d4a3aad 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-2.10 / 74af92713123574b90dac1bc60fb86bf31cae1c7 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/zulu-7-amd64:Azul Systems,
Inc.-1.7.0_262-b10 /usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.
[jira] [Commented] (HADOOP-14451) Deadlock in NativeIO
[ https://issues.apache.org/jira/browse/HADOOP-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827841#comment-17827841 ] ASF GitHub Bot commented on HADOOP-14451: - vinayakumarb merged PR #6632: URL: https://github.com/apache/hadoop/pull/6632 > Deadlock in NativeIO > > > Key: HADOOP-14451 > URL: https://issues.apache.org/jira/browse/HADOOP-14451 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Ajith S >Assignee: Vinayakumar B >Priority: Blocker > Labels: pull-request-available > Attachments: HADOOP-14451-01.patch, HADOOP-14451-02.patch, > HADOOP-14451-03.patch, HADOOP-14451-04.patch, Nodemanager.jstack > > Time Spent: 0.5h > Remaining Estimate: 0h > > * Scenario: > 1. One thread calls a static method of NativeIO, which loads static block > of NativeIo. > 2. Second thread calls a static method of NativeIo.POSIX, which loads a > static block of NativeIO.POSIX class > Both try to lock on same object inside native code gets into deadlock. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14451) Deadlock in NativeIO
[ https://issues.apache.org/jira/browse/HADOOP-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827842#comment-17827842 ] ASF GitHub Bot commented on HADOOP-14451: - vinayakumarb commented on PR #6632: URL: https://github.com/apache/hadoop/pull/6632#issuecomment-2002953187 Thanks @dineshchitlangia > Deadlock in NativeIO > > > Key: HADOOP-14451 > URL: https://issues.apache.org/jira/browse/HADOOP-14451 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Ajith S >Assignee: Vinayakumar B >Priority: Blocker > Labels: pull-request-available > Attachments: HADOOP-14451-01.patch, HADOOP-14451-02.patch, > HADOOP-14451-03.patch, HADOOP-14451-04.patch, Nodemanager.jstack > > Time Spent: 0.5h > Remaining Estimate: 0h > > * Scenario: > 1. One thread calls a static method of NativeIO, which loads static block > of NativeIo. > 2. Second thread calls a static method of NativeIo.POSIX, which loads a > static block of NativeIO.POSIX class > Both try to lock on same object inside native code gets into deadlock. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-14451. Deadlock in NativeIO [hadoop]
vinayakumarb commented on PR #6632: URL: https://github.com/apache/hadoop/pull/6632#issuecomment-2002953187 Thanks @dineshchitlangia -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-14451. Deadlock in NativeIO [hadoop]
vinayakumarb merged PR #6632: URL: https://github.com/apache/hadoop/pull/6632 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] WIP [hadoop]
hadoop-yetus commented on PR #6629:
URL: https://github.com/apache/hadoop/pull/6629#issuecomment-2002948697
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | pathlen | 0m 0s |
[/results-pathlen.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6629/5/artifact/out/results-pathlen.txt)
| The patch appears to contain 7 files with names longer than 240 |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 5s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 21s | | trunk passed |
| +1 :green_heart: | compile | 9m 58s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 8m 58s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 2m 9s | | trunk passed |
| +1 :green_heart: | mvnsite | 14m 13s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 17s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 5m 31s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 13s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 12s | | branch/hadoop-assemblies no spotbugs
output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 0m 46s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6629/5/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf-warnings.html)
| hadoop-hdfs-project/hadoop-hdfs-rbf in trunk has 1 extant spotbugs
warnings. |
| -1 :x: | spotbugs | 6m 59s |
[/branch-spotbugs-hadoop-yarn-project-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6629/5/artifact/out/branch-spotbugs-hadoop-yarn-project-warnings.html)
| hadoop-yarn-project in trunk has 1 extant spotbugs warnings. |
| -1 :x: | spotbugs | 20m 12s |
[/branch-spotbugs-root-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6629/5/artifact/out/branch-spotbugs-root-warnings.html)
| root in trunk has 5 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 21m 39s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 1m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 23m 27s | | the patch passed |
| +1 :green_heart: | compile | 9m 41s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 9m 41s | | the patch passed |
| +1 :green_heart: | compile | 8m 45s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 8m 45s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 14s | | the patch passed |
| +1 :green_heart: | mvnsite | 12m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 11s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 5m 25s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 14s | | hadoop-project has no data from
spotbugs |
| +0 :ok: | spotbugs | 0m 11s | | hadoop-assemblies has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 21m 17s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 662m 33s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6629/5/artifact/out/patch-unit-root.txt)
| root in the patch passed. |
| +1 :green_heart: | asflicense | 1m 4s | | The patch does not
generate ASF License warnings. |
| | | 935m 12s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestReadStri
Re: [PR] WIP [hadoop]
hadoop-yetus commented on PR #6629:
URL: https://github.com/apache/hadoop/pull/6629#issuecomment-2002941724
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | pathlen | 0m 0s |
[/results-pathlen.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6629/4/artifact/out/results-pathlen.txt)
| The patch appears to contain 7 files with names longer than 240 |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 9s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 46s | | trunk passed |
| +1 :green_heart: | compile | 9m 57s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 9m 12s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 2m 17s | | trunk passed |
| +1 :green_heart: | mvnsite | 13m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 58s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 4m 56s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 12s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 11s | | branch/hadoop-assemblies no spotbugs
output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 7m 10s |
[/branch-spotbugs-hadoop-yarn-project-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6629/4/artifact/out/branch-spotbugs-hadoop-yarn-project-warnings.html)
| hadoop-yarn-project in trunk has 1 extant spotbugs warnings. |
| -1 :x: | spotbugs | 20m 16s |
[/branch-spotbugs-root-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6629/4/artifact/out/branch-spotbugs-root-warnings.html)
| root in trunk has 5 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 22m 2s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 1m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 23m 19s | | the patch passed |
| +1 :green_heart: | compile | 9m 30s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 9m 30s | | the patch passed |
| +1 :green_heart: | compile | 8m 56s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 8m 56s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 6s | | the patch passed |
| +1 :green_heart: | mvnsite | 11m 41s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 54s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 5m 8s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 11s | | hadoop-project has no data from
spotbugs |
| +0 :ok: | spotbugs | 0m 11s | | hadoop-assemblies has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 22m 5s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 662m 2s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6629/4/artifact/out/patch-unit-root.txt)
| root in the patch failed. |
| +1 :green_heart: | asflicense | 1m 3s | | The patch does not
generate ASF License warnings. |
| | | 931m 25s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.TestPread |
| | hadoop.hdfs.server.namenode.TestAddStripedBlocks |
| | hadoop.hdfs.TestDFSClientFailover |
| | hadoop.hdfs.TestDecommissionWithStriped |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy |
| | hadoop.hdfs.server.balancer.TestBalancerWithHA
Re: [PR] HDFS-17299. Adding rack failure tolerance when creating a new file [hadoop]
shahrs87 merged PR #6614: URL: https://github.com/apache/hadoop/pull/6614 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17380. FsImageValidation: remove inaccessible nodes. [hadoop]
Hexiaoqiao commented on PR #6549: URL: https://github.com/apache/hadoop/pull/6549#issuecomment-2002881411 Committed to trunk. Please let me know if we need to backport to other active branches. Thanks @szetszwo for your works! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17380. FsImageValidation: remove inaccessible nodes. [hadoop]
Hexiaoqiao merged PR #6549: URL: https://github.com/apache/hadoop/pull/6549 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-19113) Compatibility Benchmark over HCFS Implementations: Improvement
[ https://issues.apache.org/jira/browse/HADOOP-19113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Han Liu reassigned HADOOP-19113: Assignee: Han Liu > Compatibility Benchmark over HCFS Implementations: Improvement > -- > > Key: HADOOP-19113 > URL: https://issues.apache.org/jira/browse/HADOOP-19113 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Han Liu >Assignee: Han Liu >Priority: Minor > > Compatibility benchmark is a tool to quickly assess availabilities of > Hadoop-Compatible File System APIs. The basic benchmark has been established, > while there are still some improvements that has not been completed. > This issue tracks these further improvements of the benchmark tool. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19085) Compatibility Benchmark over HCFS Implementations
[
https://issues.apache.org/jira/browse/HADOOP-19085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827829#comment-17827829
]
Han Liu commented on HADOOP-19085:
--
Really thanks for the code review and the final commit of the code! [~drankye]
I created a new issue for further improvements of the benchmark. Improvements
may include suggestions that has not been completed, such as PathCapabilities
in the final report, windows support, etc. We can continue the discussion and
future works there. [~ste...@apache.org] [~drankye]
Link: https://issues.apache.org/jira/browse/HADOOP-19113
> Compatibility Benchmark over HCFS Implementations
> -
>
> Key: HADOOP-19085
> URL: https://issues.apache.org/jira/browse/HADOOP-19085
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Han Liu
>Assignee: Han Liu
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-19085.001.patch, HDFS Compatibility Benchmark
> Design.pdf
>
>
> {*}Background：{*}Hadoop-Compatible File System (HCFS) is a core conception in
> big data storage ecosystem, providing unified interfaces and generally clear
> semantics, and has become the de-factor standard for industry storage systems
> to follow and conform with. There have been a series of HCFS implementations
> in Hadoop, such as S3AFileSystem for Amazon's S3 Object Store, WASB for
> Microsoft's Azure Blob Storage and OSS connector for Alibaba Cloud Object
> Storage, and more from storage service's providers on their own.
> {*}Problems：{*}However, as indicated by introduction.md, there is no formal
> suite to do compatibility assessment of a file system for all such HCFS
> implementations. Thus, whether the functionality is well accomplished and
> meets the core compatible expectations mainly relies on service provider's
> own report. Meanwhile, Hadoop is also developing and new features are
> continuously contributing to HCFS interfaces for existing implementations to
> follow and update, in which case, Hadoop also needs a tool to quickly assess
> if these features are supported or not for a specific HCFS implementation.
> Besides, the known hadoop command line tool or hdfs shell is used to directly
> interact with a HCFS storage system, where most commands correspond to
> specific HCFS interfaces and work well. Still, there are cases that are
> complicated and may not work, like expunge command. To check such commands
> for an HCFS, we also need an approach to figure them out.
> {*}Proposal：{*}Accordingly, we propose to define a formal HCFS compatibility
> benchmark and provide corresponding tool to do the compatibility assessment
> for an HCFS storage system. The benchmark and tool should consider both HCFS
> interfaces and hdfs shell commands. Different scenarios require different
> kinds of compatibilities. For such consideration, we could define different
> suites in the benchmark.
> *Benefits:* We intend the benchmark and tool to be useful for both storage
> providers and storage users. For end users, it can be used to evalute the
> compatibility level and determine if the storage system in question is
> suitable for the required scenarios. For storage providers, it helps to
> quickly generate an objective and reliable report about core functioins of
> the storage service. As an instance, if the HCFS got a 100% on a suite named
> 'tpcds', it is demonstrated that all functions needed by a tpcds program have
> been well achieved. It is also a guide indicating how storage service
> abilities can map to HCFS interfaces, such as storage class on S3.
> Any thoughts? Comments and feedback are mostly welcomed. Thanks in advance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19113) Compatibility Benchmark over HCFS Implementations: Improvement
Han Liu created HADOOP-19113: Summary: Compatibility Benchmark over HCFS Implementations: Improvement Key: HADOOP-19113 URL: https://issues.apache.org/jira/browse/HADOOP-19113 Project: Hadoop Common Issue Type: Improvement Reporter: Han Liu Compatibility benchmark is a tool to quickly assess availabilities of Hadoop-Compatible File System APIs. The basic benchmark has been established, while there are still some improvements that has not been completed. This issue tracks these further improvements of the benchmark tool. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] WIP [hadoop]
hadoop-yetus commented on PR #6629:
URL: https://github.com/apache/hadoop/pull/6629#issuecomment-2002661230
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | pathlen | 0m 0s |
[/results-pathlen.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6629/3/artifact/out/results-pathlen.txt)
| The patch appears to contain 7 files with names longer than 240 |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 41s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 36m 44s | | trunk passed |
| +1 :green_heart: | compile | 19m 33s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 17m 53s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 56s | | trunk passed |
| +1 :green_heart: | mvnsite | 21m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 9m 3s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 7m 47s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 19s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 20s | | branch/hadoop-assemblies no spotbugs
output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 11m 14s |
[/branch-spotbugs-hadoop-yarn-project-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6629/3/artifact/out/branch-spotbugs-hadoop-yarn-project-warnings.html)
| hadoop-yarn-project in trunk has 1 extant spotbugs warnings. |
| -1 :x: | spotbugs | 32m 7s |
[/branch-spotbugs-root-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6629/3/artifact/out/branch-spotbugs-root-warnings.html)
| root in trunk has 5 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 38m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 1m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 42m 40s | | the patch passed |
| +1 :green_heart: | compile | 18m 58s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 18m 58s | | the patch passed |
| +1 :green_heart: | compile | 17m 37s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 17m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 42s | | the patch passed |
| +1 :green_heart: | mvnsite | 20m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 8m 51s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 7m 50s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 19s | | hadoop-project has no data from
spotbugs |
| +0 :ok: | spotbugs | 0m 20s | | hadoop-assemblies has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 37m 57s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 777m 5s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6629/3/artifact/out/patch-unit-root.txt)
| root in the patch failed. |
| +1 :green_heart: | asflicense | 1m 39s | | The patch does not
generate ASF License warnings. |
| | | 1232m 34s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 |
| | hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6629/3/artifact/out/Dockerfile
[jira] [Commented] (HADOOP-14451) Deadlock in NativeIO
[
https://issues.apache.org/jira/browse/HADOOP-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827807#comment-17827807
]
ASF GitHub Bot commented on HADOOP-14451:
-
hadoop-yetus commented on PR #6632:
URL: https://github.com/apache/hadoop/pull/6632#issuecomment-2002568931
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 18m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 12s | | trunk passed |
| +1 :green_heart: | compile | 7m 59s | | trunk passed |
| +1 :green_heart: | checkstyle | 0m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 1s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed |
| -1 :x: | spotbugs | 1m 31s |
[/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6632/1/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html)
| hadoop-common-project/hadoop-common in trunk has 1 extant spotbugs
warnings. |
| +1 :green_heart: | shadedclient | 20m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 7m 37s | | the patch passed |
| +1 :green_heart: | cc | 7m 37s | | the patch passed |
| +1 :green_heart: | golang | 7m 37s | | the patch passed |
| +1 :green_heart: | javac | 7m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 42s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 58s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 34s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 2s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 16m 32s |
[/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6632/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 134m 8s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.crypto.TestCryptoStreamsWithOpensslSm4CtrCryptoCodec |
| | hadoop.crypto.TestCryptoCodec |
| | hadoop.service.launcher.TestServiceInterruptHandling |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6632/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6632 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc golang |
| uname | Linux 7ba18ff39419 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f5683d6fd75c74d4f93b6afdddc8ee9ff4d3a93a |
| Default Java | Red Hat, Inc.-1.8.0_402-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6632/1/testReport/ |
| Max. process+thread count | 1994 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6632/1/console |
| versions | git=2.9.5 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Deadlock in NativeIO
>
>
> Key: HADOOP-14451
> URL: https://issues.apache.org/jira/browse/HADOOP-14451
> Project: Hadoop Common
> Is
Re: [PR] HADOOP-14451. Deadlock in NativeIO [hadoop]
hadoop-yetus commented on PR #6632:
URL: https://github.com/apache/hadoop/pull/6632#issuecomment-2002568931
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 18m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 12s | | trunk passed |
| +1 :green_heart: | compile | 7m 59s | | trunk passed |
| +1 :green_heart: | checkstyle | 0m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 1s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed |
| -1 :x: | spotbugs | 1m 31s |
[/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6632/1/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html)
| hadoop-common-project/hadoop-common in trunk has 1 extant spotbugs
warnings. |
| +1 :green_heart: | shadedclient | 20m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 7m 37s | | the patch passed |
| +1 :green_heart: | cc | 7m 37s | | the patch passed |
| +1 :green_heart: | golang | 7m 37s | | the patch passed |
| +1 :green_heart: | javac | 7m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 42s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 58s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 34s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 2s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 16m 32s |
[/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6632/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 134m 8s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.crypto.TestCryptoStreamsWithOpensslSm4CtrCryptoCodec |
| | hadoop.crypto.TestCryptoCodec |
| | hadoop.service.launcher.TestServiceInterruptHandling |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6632/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6632 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc golang |
| uname | Linux 7ba18ff39419 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f5683d6fd75c74d4f93b6afdddc8ee9ff4d3a93a |
| Default Java | Red Hat, Inc.-1.8.0_402-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6632/1/testReport/ |
| Max. process+thread count | 1994 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6632/1/console |
| versions | git=2.9.5 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For ad
Re: [PR] HDFS-17299. Adding rack failure tolerance when creating a new file [hadoop]
shahrs87 commented on code in PR #6614:
URL: https://github.com/apache/hadoop/pull/6614#discussion_r1527571968
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java:
##
@@ -1707,4 +1708,154 @@ public void testStorageFavouredNodes()
assertEquals("Number of SSD should be 1 but was : " + numSSD, 1, numSSD);
}
}
+
+ @Test
+ public void testSingleRackFailureDuringPipelineSetupMinReplicationPossible()
throws Exception {
+Configuration conf = getTestConfiguration();
+conf.setClass(
+DFSConfigKeys.DFS_BLOCK_REPLICATOR_CLASSNAME_KEY,
+BlockPlacementPolicyRackFaultTolerant.class,
+BlockPlacementPolicy.class);
+conf.setBoolean(
+HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.ENABLE_KEY,
+false);
+conf.setInt(HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.
+MIN_REPLICATION, 2);
+// 3 racks & 6 nodes. 1 per rack for 2 racks and 3 nodes in the 3rd rack
+try (MiniDFSCluster cluster = new
MiniDFSCluster.Builder(conf).numDataNodes(6)
+.racks(new String[] {"/rack1", "/rack2", "/rack3", "/rack3", "/rack3",
"/rack3"}).build()) {
+ cluster.waitClusterUp();
+ DistributedFileSystem fs = cluster.getFileSystem();
+ // kill all the DNs in the 3rd rack.
+ cluster.stopDataNode(5);
+ cluster.stopDataNode(4);
+ cluster.stopDataNode(3);
+ cluster.stopDataNode(2);
+
+ // create a file with replication 3, for rack fault tolerant BPP,
+ DFSTestUtil.createFile(fs, new Path("/testFile"), 1024L, (short) 3,
1024L);
+}
+ }
+
+ @Test
+ public void
testSingleRackFailureDuringPipelineSetupMinReplicationImpossible()
+ throws Exception {
+Configuration conf = getTestConfiguration();
+conf.setClass(DFSConfigKeys.DFS_BLOCK_REPLICATOR_CLASSNAME_KEY,
+BlockPlacementPolicyRackFaultTolerant.class,
BlockPlacementPolicy.class);
+
conf.setBoolean(HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.ENABLE_KEY,
false);
+
conf.setInt(HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.MIN_REPLICATION,
3);
+// 3 racks & 6 nodes. 1 per rack for 2 racks and 3 nodes in the 3rd rack
+try (MiniDFSCluster cluster = new
MiniDFSCluster.Builder(conf).numDataNodes(6)
+.racks(new String[] {"/rack1", "/rack2", "/rack3", "/rack3", "/rack3",
"/rack3"}).build()) {
+ cluster.waitClusterUp();
+ DistributedFileSystem fs = cluster.getFileSystem();
+ // kill one DN, so only 2 racks stays with active DN
+ cluster.stopDataNode(5);
+ cluster.stopDataNode(4);
+ cluster.stopDataNode(3);
+ cluster.stopDataNode(2);
+ boolean threw = false;
+ try {
+DFSTestUtil.createFile(fs, new Path("/testFile"), 1024L, (short) 3,
1024L);
+ } catch (IOException e) {
+threw = true;
+ }
+ assertTrue(threw);
+}
+ }
+
+ @Test
+ public void
testMultipleRackFailureDuringPipelineSetupMinReplicationPossible() throws
Exception {
+Configuration conf = getTestConfiguration();
+conf.setClass(
+DFSConfigKeys.DFS_BLOCK_REPLICATOR_CLASSNAME_KEY,
+BlockPlacementPolicyRackFaultTolerant.class,
+BlockPlacementPolicy.class);
+conf.setBoolean(
+HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.ENABLE_KEY,
+false);
+conf.setInt(HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.
+MIN_REPLICATION, 1);
+// 3 racks & 3 nodes. 1 per rack
Review Comment:
The comments are not correct. Can you please update them? @ritegarg
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java:
##
@@ -1707,4 +1708,154 @@ public void testStorageFavouredNodes()
assertEquals("Number of SSD should be 1 but was : " + numSSD, 1, numSSD);
}
}
+
+ @Test
+ public void testSingleRackFailureDuringPipelineSetupMinReplicationPossible()
throws Exception {
+Configuration conf = getTestConfiguration();
+conf.setClass(
+DFSConfigKeys.DFS_BLOCK_REPLICATOR_CLASSNAME_KEY,
+BlockPlacementPolicyRackFaultTolerant.class,
+BlockPlacementPolicy.class);
+conf.setBoolean(
+HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.ENABLE_KEY,
+false);
+conf.setInt(HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.
+MIN_REPLICATION, 2);
+// 3 racks & 6 nodes. 1 per rack for 2 racks and 3 nodes in the 3rd rack
+try (MiniDFSCluster cluster = new
MiniDFSCluster.Builder(conf).numDataNodes(6)
+.racks(new String[] {"/rack1", "/rack2", "/rack3", "/rack3", "/rack3",
"/rack3"}).build()) {
+ cluster.waitClusterUp();
+ DistributedFileSystem fs = cluster.getFileSystem();
+ // kill all the DNs in the 3rd rack.
+ cluster.stopDataNode(5);
+ cluster.stopDataNode(4);
+ cluster.stopDataNode(3);
+ cluster.stopDataNode(2);
+
+
Re: [PR] YARN-11664: Remove HDFS Binaries/Jars Dependency From Yarn [hadoop]
hadoop-yetus commented on PR #6631:
URL: https://github.com/apache/hadoop/pull/6631#issuecomment-2002537054
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 2s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 15 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 23s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 15s | | trunk passed |
| +1 :green_heart: | compile | 17m 19s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 15m 58s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 59s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 7m 16s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 2m 36s |
[/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6631/2/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html)
| hadoop-common-project/hadoop-common in trunk has 1 extant spotbugs
warnings. |
| -1 :x: | spotbugs | 2m 43s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6631/2/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html)
| hadoop-hdfs-project/hadoop-hdfs-client in trunk has 1 extant spotbugs
warnings. |
| -1 :x: | spotbugs | 1m 32s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6631/2/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf-warnings.html)
| hadoop-hdfs-project/hadoop-hdfs-rbf in trunk has 1 extant spotbugs
warnings. |
| -1 :x: | spotbugs | 1m 13s |
[/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services_hadoop-yarn-services-core-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6631/2/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services_hadoop-yarn-services-core-warnings.html)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core
in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 33m 8s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 5m 10s | | the patch passed |
| +1 :green_heart: | compile | 16m 47s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 16m 47s | | the patch passed |
| +1 :green_heart: | compile | 16m 8s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 16m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 25s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6631/2/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 7 new + 1212 unchanged - 5 fixed = 1219 total
(was 1217) |
| +1 :green_heart: | mvnsite | 8m 23s | | the patch passed |
| -1 :x: | javadoc | 0m 59s |
[/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6631/2/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| hadoop-hdfs-client in the patch failed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| +1 :green_heart: | javadoc | 7m 13s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 16m 19s | | the patch passed |
| +1 :gree
[jira] [Commented] (HADOOP-14451) Deadlock in NativeIO
[ https://issues.apache.org/jira/browse/HADOOP-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827793#comment-17827793 ] ASF GitHub Bot commented on HADOOP-14451: - vinayakumarb opened a new pull request, #6632: URL: https://github.com/apache/hadoop/pull/6632 ### Description of PR * Scenario: 1. One thread calls a static method of NativeIO, which loads static block of NativeIo. 2. Second thread calls a static method of NativeIo.POSIX, which loads a static block of NativeIO.POSIX class Both try to lock on same object inside native code gets into deadlock. ### How was this patch tested? Reproducible Unit tests. ### For code changes: - [* ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > Deadlock in NativeIO > > > Key: HADOOP-14451 > URL: https://issues.apache.org/jira/browse/HADOOP-14451 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Ajith S >Assignee: Vinayakumar B >Priority: Blocker > Labels: pull-request-available > Attachments: HADOOP-14451-01.patch, HADOOP-14451-02.patch, > HADOOP-14451-03.patch, HADOOP-14451-04.patch, Nodemanager.jstack > > Time Spent: 0.5h > Remaining Estimate: 0h > > * Scenario: > 1. One thread calls a static method of NativeIO, which loads static block > of NativeIo. > 2. Second thread calls a static method of NativeIo.POSIX, which loads a > static block of NativeIO.POSIX class > Both try to lock on same object inside native code gets into deadlock. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] HADOOP-14451. Deadlock in NativeIO [hadoop]
vinayakumarb opened a new pull request, #6632: URL: https://github.com/apache/hadoop/pull/6632 ### Description of PR * Scenario: 1. One thread calls a static method of NativeIO, which loads static block of NativeIo. 2. Second thread calls a static method of NativeIo.POSIX, which loads a static block of NativeIO.POSIX class Both try to lock on same object inside native code gets into deadlock. ### How was this patch tested? Reproducible Unit tests. ### For code changes: - [* ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14451) Deadlock in NativeIO
[ https://issues.apache.org/jira/browse/HADOOP-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HADOOP-14451: --- Description: * Scenario: 1. One thread calls a static method of NativeIO, which loads static block of NativeIo. 2. Second thread calls a static method of NativeIo.POSIX, which loads a static block of NativeIO.POSIX class Both try to lock on same object inside native code gets into deadlock. > Deadlock in NativeIO > > > Key: HADOOP-14451 > URL: https://issues.apache.org/jira/browse/HADOOP-14451 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Ajith S >Assignee: Vinayakumar B >Priority: Blocker > Labels: pull-request-available > Attachments: HADOOP-14451-01.patch, HADOOP-14451-02.patch, > HADOOP-14451-03.patch, HADOOP-14451-04.patch, Nodemanager.jstack > > Time Spent: 0.5h > Remaining Estimate: 0h > > * Scenario: > 1. One thread calls a static method of NativeIO, which loads static block > of NativeIo. > 2. Second thread calls a static method of NativeIo.POSIX, which loads a > static block of NativeIO.POSIX class > Both try to lock on same object inside native code gets into deadlock. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19085) Compatibility Benchmark over HCFS Implementations
[
https://issues.apache.org/jira/browse/HADOOP-19085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng resolved HADOOP-19085.
Hadoop Flags: Reviewed
Target Version/s: 3.3.6
Resolution: Fixed
> Compatibility Benchmark over HCFS Implementations
> -
>
> Key: HADOOP-19085
> URL: https://issues.apache.org/jira/browse/HADOOP-19085
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Han Liu
>Assignee: Han Liu
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-19085.001.patch, HDFS Compatibility Benchmark
> Design.pdf
>
>
> {*}Background：{*}Hadoop-Compatible File System (HCFS) is a core conception in
> big data storage ecosystem, providing unified interfaces and generally clear
> semantics, and has become the de-factor standard for industry storage systems
> to follow and conform with. There have been a series of HCFS implementations
> in Hadoop, such as S3AFileSystem for Amazon's S3 Object Store, WASB for
> Microsoft's Azure Blob Storage and OSS connector for Alibaba Cloud Object
> Storage, and more from storage service's providers on their own.
> {*}Problems：{*}However, as indicated by introduction.md, there is no formal
> suite to do compatibility assessment of a file system for all such HCFS
> implementations. Thus, whether the functionality is well accomplished and
> meets the core compatible expectations mainly relies on service provider's
> own report. Meanwhile, Hadoop is also developing and new features are
> continuously contributing to HCFS interfaces for existing implementations to
> follow and update, in which case, Hadoop also needs a tool to quickly assess
> if these features are supported or not for a specific HCFS implementation.
> Besides, the known hadoop command line tool or hdfs shell is used to directly
> interact with a HCFS storage system, where most commands correspond to
> specific HCFS interfaces and work well. Still, there are cases that are
> complicated and may not work, like expunge command. To check such commands
> for an HCFS, we also need an approach to figure them out.
> {*}Proposal：{*}Accordingly, we propose to define a formal HCFS compatibility
> benchmark and provide corresponding tool to do the compatibility assessment
> for an HCFS storage system. The benchmark and tool should consider both HCFS
> interfaces and hdfs shell commands. Different scenarios require different
> kinds of compatibilities. For such consideration, we could define different
> suites in the benchmark.
> *Benefits:* We intend the benchmark and tool to be useful for both storage
> providers and storage users. For end users, it can be used to evalute the
> compatibility level and determine if the storage system in question is
> suitable for the required scenarios. For storage providers, it helps to
> quickly generate an objective and reliable report about core functioins of
> the storage service. As an instance, if the HCFS got a 100% on a suite named
> 'tpcds', it is demonstrated that all functions needed by a tpcds program have
> been well achieved. It is also a guide indicating how storage service
> abilities can map to HCFS interfaces, such as storage class on S3.
> Any thoughts? Comments and feedback are mostly welcomed. Thanks in advance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19085) Compatibility Benchmark over HCFS Implementations
[
https://issues.apache.org/jira/browse/HADOOP-19085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827774#comment-17827774
]
Kai Zheng commented on HADOOP-19085:
For the valuable suggestions mentioned in the discussion, I suggest we follow
up in separate issues, [~han.liu].
> Compatibility Benchmark over HCFS Implementations
> -
>
> Key: HADOOP-19085
> URL: https://issues.apache.org/jira/browse/HADOOP-19085
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Han Liu
>Assignee: Han Liu
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-19085.001.patch, HDFS Compatibility Benchmark
> Design.pdf
>
>
> {*}Background：{*}Hadoop-Compatible File System (HCFS) is a core conception in
> big data storage ecosystem, providing unified interfaces and generally clear
> semantics, and has become the de-factor standard for industry storage systems
> to follow and conform with. There have been a series of HCFS implementations
> in Hadoop, such as S3AFileSystem for Amazon's S3 Object Store, WASB for
> Microsoft's Azure Blob Storage and OSS connector for Alibaba Cloud Object
> Storage, and more from storage service's providers on their own.
> {*}Problems：{*}However, as indicated by introduction.md, there is no formal
> suite to do compatibility assessment of a file system for all such HCFS
> implementations. Thus, whether the functionality is well accomplished and
> meets the core compatible expectations mainly relies on service provider's
> own report. Meanwhile, Hadoop is also developing and new features are
> continuously contributing to HCFS interfaces for existing implementations to
> follow and update, in which case, Hadoop also needs a tool to quickly assess
> if these features are supported or not for a specific HCFS implementation.
> Besides, the known hadoop command line tool or hdfs shell is used to directly
> interact with a HCFS storage system, where most commands correspond to
> specific HCFS interfaces and work well. Still, there are cases that are
> complicated and may not work, like expunge command. To check such commands
> for an HCFS, we also need an approach to figure them out.
> {*}Proposal：{*}Accordingly, we propose to define a formal HCFS compatibility
> benchmark and provide corresponding tool to do the compatibility assessment
> for an HCFS storage system. The benchmark and tool should consider both HCFS
> interfaces and hdfs shell commands. Different scenarios require different
> kinds of compatibilities. For such consideration, we could define different
> suites in the benchmark.
> *Benefits:* We intend the benchmark and tool to be useful for both storage
> providers and storage users. For end users, it can be used to evalute the
> compatibility level and determine if the storage system in question is
> suitable for the required scenarios. For storage providers, it helps to
> quickly generate an objective and reliable report about core functioins of
> the storage service. As an instance, if the HCFS got a 100% on a suite named
> 'tpcds', it is demonstrated that all functions needed by a tpcds program have
> been well achieved. It is also a guide indicating how storage service
> abilities can map to HCFS interfaces, such as storage class on S3.
> Any thoughts? Comments and feedback are mostly welcomed. Thanks in advance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19085) Compatibility Benchmark over HCFS Implementations
[
https://issues.apache.org/jira/browse/HADOOP-19085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827773#comment-17827773
]
Kai Zheng commented on HADOOP-19085:
The patch was just committed. Thanks [~han.liu] for the contribution, and
[~ste...@apache.org] for the guidance!
> Compatibility Benchmark over HCFS Implementations
> -
>
> Key: HADOOP-19085
> URL: https://issues.apache.org/jira/browse/HADOOP-19085
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Han Liu
>Assignee: Han Liu
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-19085.001.patch, HDFS Compatibility Benchmark
> Design.pdf
>
>
> {*}Background：{*}Hadoop-Compatible File System (HCFS) is a core conception in
> big data storage ecosystem, providing unified interfaces and generally clear
> semantics, and has become the de-factor standard for industry storage systems
> to follow and conform with. There have been a series of HCFS implementations
> in Hadoop, such as S3AFileSystem for Amazon's S3 Object Store, WASB for
> Microsoft's Azure Blob Storage and OSS connector for Alibaba Cloud Object
> Storage, and more from storage service's providers on their own.
> {*}Problems：{*}However, as indicated by introduction.md, there is no formal
> suite to do compatibility assessment of a file system for all such HCFS
> implementations. Thus, whether the functionality is well accomplished and
> meets the core compatible expectations mainly relies on service provider's
> own report. Meanwhile, Hadoop is also developing and new features are
> continuously contributing to HCFS interfaces for existing implementations to
> follow and update, in which case, Hadoop also needs a tool to quickly assess
> if these features are supported or not for a specific HCFS implementation.
> Besides, the known hadoop command line tool or hdfs shell is used to directly
> interact with a HCFS storage system, where most commands correspond to
> specific HCFS interfaces and work well. Still, there are cases that are
> complicated and may not work, like expunge command. To check such commands
> for an HCFS, we also need an approach to figure them out.
> {*}Proposal：{*}Accordingly, we propose to define a formal HCFS compatibility
> benchmark and provide corresponding tool to do the compatibility assessment
> for an HCFS storage system. The benchmark and tool should consider both HCFS
> interfaces and hdfs shell commands. Different scenarios require different
> kinds of compatibilities. For such consideration, we could define different
> suites in the benchmark.
> *Benefits:* We intend the benchmark and tool to be useful for both storage
> providers and storage users. For end users, it can be used to evalute the
> compatibility level and determine if the storage system in question is
> suitable for the required scenarios. For storage providers, it helps to
> quickly generate an objective and reliable report about core functioins of
> the storage service. As an instance, if the HCFS got a 100% on a suite named
> 'tpcds', it is demonstrated that all functions needed by a tpcds program have
> been well achieved. It is also a guide indicating how storage service
> abilities can map to HCFS interfaces, such as storage class on S3.
> Any thoughts? Comments and feedback are mostly welcomed. Thanks in advance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-19112) Hadoop 3.4.0 release wrap-up
[ https://issues.apache.org/jira/browse/HADOOP-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan reassigned HADOOP-19112: --- Assignee: Shilun Fan > Hadoop 3.4.0 release wrap-up > > > Key: HADOOP-19112 > URL: https://issues.apache.org/jira/browse/HADOOP-19112 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19112) Hadoop 3.4.0 release wrap-up
Shilun Fan created HADOOP-19112: --- Summary: Hadoop 3.4.0 release wrap-up Key: HADOOP-19112 URL: https://issues.apache.org/jira/browse/HADOOP-19112 Project: Hadoop Common Issue Type: Sub-task Components: common Affects Versions: 3.4.0 Reporter: Shilun Fan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
