[jira] [Commented] (HDFS-16939) Fix the thread safety bug in LowRedundancyBlocks
[ https://issues.apache.org/jira/browse/HDFS-16939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17696049#comment-17696049 ] ASF GitHub Bot commented on HDFS-16939: --- slfan1989 commented on PR #5450: URL: https://github.com/apache/hadoop/pull/5450#issuecomment-1453082018 LGTM. > Fix the thread safety bug in LowRedundancyBlocks > > > Key: HDFS-16939 > URL: https://issues.apache.org/jira/browse/HDFS-16939 > Project: Hadoop HDFS > Issue Type: Bug > Components: namanode >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > > The remove method in LowRedundancyBlocks is not protected by synchronized. > This method is private and is called by BlockManager. As a result, > priorityQueues has the risk of being accessed concurrently by multiple > threads. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16939) Fix the thread safety bug in LowRedundancyBlocks
[ https://issues.apache.org/jira/browse/HDFS-16939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17696040#comment-17696040 ] ASF GitHub Bot commented on HDFS-16939: --- hadoop-yetus commented on PR #5450: URL: https://github.com/apache/hadoop/pull/5450#issuecomment-1453058032 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | docker | 1m 9s | | Docker failed to build run-specific yetus/hadoop:tp-3985}. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/5450 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5450/1/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. > Fix the thread safety bug in LowRedundancyBlocks > > > Key: HDFS-16939 > URL: https://issues.apache.org/jira/browse/HDFS-16939 > Project: Hadoop HDFS > Issue Type: Bug > Components: namanode >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > > The remove method in LowRedundancyBlocks is not protected by synchronized. > This method is private and is called by BlockManager. As a result, > priorityQueues has the risk of being accessed concurrently by multiple > threads. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16939) Fix the thread safety bug in LowRedundancyBlocks
[ https://issues.apache.org/jira/browse/HDFS-16939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16939: -- Labels: pull-request-available (was: ) > Fix the thread safety bug in LowRedundancyBlocks > > > Key: HDFS-16939 > URL: https://issues.apache.org/jira/browse/HDFS-16939 > Project: Hadoop HDFS > Issue Type: Bug > Components: namanode >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > > The remove method in LowRedundancyBlocks is not protected by synchronized. > This method is private and is called by BlockManager. As a result, > priorityQueues has the risk of being accessed concurrently by multiple > threads. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16939) Fix the thread safety bug in LowRedundancyBlocks
[ https://issues.apache.org/jira/browse/HDFS-16939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17696038#comment-17696038 ] ASF GitHub Bot commented on HDFS-16939: --- zhangshuyan0 opened a new pull request, #5450: URL: https://github.com/apache/hadoop/pull/5450 The remove method in LowRedundancyBlocks is not protected by synchronized. This method is private and is called by BlockManager. As a result, priorityQueues has the risk of being accessed concurrently by multiple threads. > Fix the thread safety bug in LowRedundancyBlocks > > > Key: HDFS-16939 > URL: https://issues.apache.org/jira/browse/HDFS-16939 > Project: Hadoop HDFS > Issue Type: Bug > Components: namanode >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > > The remove method in LowRedundancyBlocks is not protected by synchronized. > This method is private and is called by BlockManager. As a result, > priorityQueues has the risk of being accessed concurrently by multiple > threads. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16939) Fix the thread safety bug in LowRedundancyBlocks
Shuyan Zhang created HDFS-16939: --- Summary: Fix the thread safety bug in LowRedundancyBlocks Key: HDFS-16939 URL: https://issues.apache.org/jira/browse/HDFS-16939 Project: Hadoop HDFS Issue Type: Bug Components: namanode Reporter: Shuyan Zhang Assignee: Shuyan Zhang The remove method in LowRedundancyBlocks is not protected by synchronized. This method is private and is called by BlockManager. As a result, priorityQueues has the risk of being accessed concurrently by multiple threads. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16931) Observer nn delete blocks asynchronously when tail OP_DELETE editlog
[ https://issues.apache.org/jira/browse/HDFS-16931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kinit updated HDFS-16931: - Priority: Major (was: Minor) > Observer nn delete blocks asynchronously when tail OP_DELETE editlog > > > Key: HDFS-16931 > URL: https://issues.apache.org/jira/browse/HDFS-16931 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: kinit >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16938) Utility to trigger heartbeat and wait until BP thread queue is fully processed
[
https://issues.apache.org/jira/browse/HDFS-16938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17696001#comment-17696001
]
ASF GitHub Bot commented on HDFS-16938:
---
saxenapranav commented on code in PR #5445:
URL: https://github.com/apache/hadoop/pull/5445#discussion_r1124018756
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ErrorReportAction.java:
##
@@ -34,7 +34,8 @@ public class ErrorReportAction implements
BPServiceActorAction {
final int errorCode;
final String errorMessage;
-
+ private boolean isReportSuccessfullySent = false;
Review Comment:
Yes. Agree with you.
> Utility to trigger heartbeat and wait until BP thread queue is fully processed
> --
>
> Key: HDFS-16938
> URL: https://issues.apache.org/jira/browse/HDFS-16938
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> As a follow-up to HDFS-16935, we should provide utility to trigger heartbeat
> and wait until BP thread queue is fully processed. This would ensure 100%
> consistency w.r.t active namenode being able to receive bad block reports
> from the given datanode. This utility would resolve flakes for the tests that
> rely on namenode's awareness of the reported bad blocks by datanodes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15765) Add support for Kerberos and Basic Auth in webhdfs
[
https://issues.apache.org/jira/browse/HDFS-15765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17695966#comment-17695966
]
ASF GitHub Bot commented on HDFS-15765:
---
hadoop-yetus commented on PR #5447:
URL: https://github.com/apache/hadoop/pull/5447#issuecomment-1452859975
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 19m 18s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 41s | | trunk passed |
| +1 :green_heart: | compile | 6m 3s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 5m 43s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 1m 18s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 50s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 2m 20s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 5m 57s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 32s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 7s | | the patch passed |
| +1 :green_heart: | compile | 5m 50s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 5m 50s | | the patch passed |
| +1 :green_heart: | compile | 5m 36s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | javac | 5m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 5s |
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5447/2/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
| hadoop-hdfs-project: The patch generated 1 new + 87 unchanged - 0 fixed =
88 total (was 87) |
| +1 :green_heart: | mvnsite | 2m 11s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 27s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 2m 3s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 5m 56s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 27s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 29s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | unit | 206m 6s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not
generate ASF License warnings. |
| | | 350m 45s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5447/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5447 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 62e3e8274ff4 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ee843013b36a761d3050c69e008f6bd234ae6985 |
| Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
[jira] [Commented] (HDFS-16938) Utility to trigger heartbeat and wait until BP thread queue is fully processed
[
https://issues.apache.org/jira/browse/HDFS-16938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17695859#comment-17695859
]
ASF GitHub Bot commented on HDFS-16938:
---
virajjasani commented on code in PR #5445:
URL: https://github.com/apache/hadoop/pull/5445#discussion_r1123555498
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ErrorReportAction.java:
##
@@ -34,7 +34,8 @@ public class ErrorReportAction implements
BPServiceActorAction {
final int errorCode;
final String errorMessage;
-
+ private boolean isReportSuccessfullySent = false;
Review Comment:
Sure, we can keep it volatile to be a bit more aggressive. However,
regardless of whichever thread calls `isReportSuccessfullySent`, the eventual
value would come as true if the action was completed, and the nature of the
utility method is to wait for it to be true so we should be good IMO. In fact,
the thread that calls it is definitely other thread only (main thread of the
test, in this case).
On the other hand, if we were to have a utility that needs strong
consistency for the value returned by `isReportSuccessfullySent` and not have
any retry/wait sort of mechanism in place, then yes volatile would be the right
choice.
Does it sound good?
> Utility to trigger heartbeat and wait until BP thread queue is fully processed
> --
>
> Key: HDFS-16938
> URL: https://issues.apache.org/jira/browse/HDFS-16938
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> As a follow-up to HDFS-16935, we should provide utility to trigger heartbeat
> and wait until BP thread queue is fully processed. This would ensure 100%
> consistency w.r.t active namenode being able to receive bad block reports
> from the given datanode. This utility would resolve flakes for the tests that
> rely on namenode's awareness of the reported bad blocks by datanodes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16938) Utility to trigger heartbeat and wait until BP thread queue is fully processed
[
https://issues.apache.org/jira/browse/HDFS-16938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17695836#comment-17695836
]
ASF GitHub Bot commented on HDFS-16938:
---
saxenapranav commented on code in PR #5445:
URL: https://github.com/apache/hadoop/pull/5445#discussion_r1123444698
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ErrorReportAction.java:
##
@@ -34,7 +34,8 @@ public class ErrorReportAction implements
BPServiceActorAction {
final int errorCode;
final String errorMessage;
-
+ private boolean isReportSuccessfullySent = false;
Review Comment:
Would volatile boolean be better? Reason being, isReportSuccessfullySent
might be called by some other thread.
> Utility to trigger heartbeat and wait until BP thread queue is fully processed
> --
>
> Key: HDFS-16938
> URL: https://issues.apache.org/jira/browse/HDFS-16938
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> As a follow-up to HDFS-16935, we should provide utility to trigger heartbeat
> and wait until BP thread queue is fully processed. This would ensure 100%
> consistency w.r.t active namenode being able to receive bad block reports
> from the given datanode. This utility would resolve flakes for the tests that
> rely on namenode's awareness of the reported bad blocks by datanodes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16938) Utility to trigger heartbeat and wait until BP thread queue is fully processed
[
https://issues.apache.org/jira/browse/HDFS-16938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17695826#comment-17695826
]
ASF GitHub Bot commented on HDFS-16938:
---
saxenapranav commented on code in PR #5445:
URL: https://github.com/apache/hadoop/pull/5445#discussion_r1123444698
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ErrorReportAction.java:
##
@@ -34,7 +34,8 @@ public class ErrorReportAction implements
BPServiceActorAction {
final int errorCode;
final String errorMessage;
-
+ private boolean isReportSuccessfullySent = false;
Review Comment:
Would atomicBoolean be better? Reason being, isReportSuccessfullySent might
be called by some other thread.
> Utility to trigger heartbeat and wait until BP thread queue is fully processed
> --
>
> Key: HDFS-16938
> URL: https://issues.apache.org/jira/browse/HDFS-16938
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> As a follow-up to HDFS-16935, we should provide utility to trigger heartbeat
> and wait until BP thread queue is fully processed. This would ensure 100%
> consistency w.r.t active namenode being able to receive bad block reports
> from the given datanode. This utility would resolve flakes for the tests that
> rely on namenode's awareness of the reported bad blocks by datanodes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15765) Add support for Kerberos and Basic Auth in webhdfs
[
https://issues.apache.org/jira/browse/HDFS-15765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17695728#comment-17695728
]
ASF GitHub Bot commented on HDFS-15765:
---
hadoop-yetus commented on PR #5447:
URL: https://github.com/apache/hadoop/pull/5447#issuecomment-1451907434
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 6s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 53s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 44s | | trunk passed |
| +1 :green_heart: | compile | 6m 38s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 6m 15s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 1m 20s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 44s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 2m 9s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 6m 13s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 3s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 5s | | the patch passed |
| +1 :green_heart: | compile | 6m 39s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 6m 39s | | the patch passed |
| +1 :green_heart: | compile | 6m 7s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | javac | 6m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 9s |
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5447/1/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
| hadoop-hdfs-project: The patch generated 1 new + 87 unchanged - 0 fixed =
88 total (was 87) |
| +1 :green_heart: | mvnsite | 2m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 27s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 2m 0s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| -1 :x: | spotbugs | 2m 44s |
[/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5447/1/artifact/out/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.html)
| hadoop-hdfs-project/hadoop-hdfs-client generated 1 new + 0 unchanged - 0
fixed = 1 total (was 0) |
| +1 :green_heart: | shadedclient | 25m 50s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 23s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | unit | 226m 20s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 394m 52s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs-client |
| | Found reliance on default encoding in
org.apache.hadoop.hdfs.web.BasicAuthConfigurator.configure(HttpURLConnection):in
org.apache.hadoop.hdfs.web.BasicAuthConfigurator.configure(HttpURLConnection):
String.getBytes() At BasicAuthConfigurator.java:[line 55] |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5447/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5447 |
| Optional Tests | dupname
[jira] [Updated] (HDFS-15765) Add support for Kerberos and Basic Auth in webhdfs
[ https://issues.apache.org/jira/browse/HDFS-15765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-15765: -- Labels: pull-request-available (was: ) > Add support for Kerberos and Basic Auth in webhdfs > -- > > Key: HDFS-15765 > URL: https://issues.apache.org/jira/browse/HDFS-15765 > Project: Hadoop HDFS > Issue Type: Bug > Components: hadoop-client >Reporter: Pushpendra Singh >Priority: Minor > Labels: pull-request-available > > webhdfs's HTTP operation like get ( GetOpParam.java) operation and other HTTP > operation has 'requireAuth' set to false and expected to work with Delegation > token only. However, when working with webhdfs over Apache Knox, delegation > token authentication is not supported, we should support Kerberos > authentication (SPNEGO) or Basic authentication for WebHdfsFileSystem if user > turns on a configuration. > Further webhdfs (WebHDFSFileSystem.java) is calling 'public URLConnection > openConnection(URL url)' and providing no way to use the kerberos > authentication, if configured. > Even after setting the UserGroupInformation with user name and keytab, > openConnection is not using the keytab for authentication. > Also WebHdfsFileSystem doesn't provide any support for HTTP BASIC > authentication (username/password). Provide support to read the password via > environment variable. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16934) org.apache.hadoop.hdfs.tools.TestDFSAdmin#testAllDatanodesReconfig regression
[
https://issues.apache.org/jira/browse/HDFS-16934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17695678#comment-17695678
]
ASF GitHub Bot commented on HDFS-16934:
---
slfan1989 commented on PR #5434:
URL: https://github.com/apache/hadoop/pull/5434#issuecomment-1451744187
> checkstyle failing on an unused import; other than that I'm happy -the
assertj collection asserts are so much better for asserting on collections and
debugging their failures.
>
> +1 pending the checkstyle fix
Thank you very much for your help to review pr! I will continue to use
`assertj` to complete the unit test.
> org.apache.hadoop.hdfs.tools.TestDFSAdmin#testAllDatanodesReconfig regression
> -
>
> Key: HDFS-16934
> URL: https://issues.apache.org/jira/browse/HDFS-16934
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: dfsadmin, test
>Affects Versions: 3.4.0, 3.3.5, 3.3.9
>Reporter: Steve Loughran
>Assignee: Shilun Fan
>Priority: Minor
> Labels: pull-request-available
>
> jenkins test failure as the logged output is in the wrong order for the
> assertions. HDFS-16624 flipped the order...without that this would have
> worked.
> {code}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at
> org.apache.hadoop.hdfs.tools.TestDFSAdmin.testAllDatanodesReconfig(TestDFSAdmin.java:1149)
> {code}
> Here the code is asserting about the contents of the output,
> {code}
> assertTrue(outs.get(0).startsWith("Reconfiguring status for node"));
> assertTrue("SUCCESS: Changed property
> dfs.datanode.peer.stats.enabled".equals(outs.get(2))
> || "SUCCESS: Changed property
> dfs.datanode.peer.stats.enabled".equals(outs.get(1))); // here
> assertTrue("\tFrom: \"false\"".equals(outs.get(3)) || "\tFrom:
> \"false\"".equals(outs.get(2)));
> assertTrue("\tTo: \"true\"".equals(outs.get(4)) || "\tTo:
> \"true\"".equals(outs.get(3)))
> {code}
> If you look at the log, the actual line is appearing in that list, just in a
> different place. race condition
> {code}
> 2023-02-24 01:02:06,275 [Listener at localhost/41795] INFO
> tools.TestDFSAdmin (TestDFSAdmin.java:testAllDatanodesReconfig(1146)) -
> dfsadmin -status -livenodes output:
> 2023-02-24 01:02:06,276 [Listener at localhost/41795] INFO
> tools.TestDFSAdmin
> (TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) - Reconfiguring
> status for node [127.0.0.1:41795]: started at Fri Feb 24 01:02:03 GMT 2023
> and finished at Fri Feb 24 01:02:03 GMT 2023.
> 2023-02-24 01:02:06,276 [Listener at localhost/41795] INFO
> tools.TestDFSAdmin
> (TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) - Reconfiguring
> status for node [127.0.0.1:34007]: started at Fri Feb 24 01:02:03 GMT
> 2023SUCCESS: Changed property dfs.datanode.peer.stats.enabled
> 2023-02-24 01:02:06,277 [Listener at localhost/41795] INFO
> tools.TestDFSAdmin
> (TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) - From: "false"
> 2023-02-24 01:02:06,277 [Listener at localhost/41795] INFO
> tools.TestDFSAdmin
> (TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) - To: "true"
> 2023-02-24 01:02:06,277 [Listener at localhost/41795] INFO
> tools.TestDFSAdmin
> (TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) - and finished
> at Fri Feb 24 01:02:03 GMT 2023.
> 2023-02-24 01:02:06,277 [Listener at localhost/41795] INFO
> tools.TestDFSAdmin
> (TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) - SUCCESS:
> Changed property dfs.datanode.peer.stats.enabled
> {code}
> we have a race condition in output generation and the assertions are clearly
> too brittle
> for the 3.3.5 release I'm not going to make this a blocker. What i will do is
> propose that the asserts move to assertJ with an assertion that the
> collection "containsExactlyInAnyOrder" all the strings.
> That will
> 1. not be brittle.
> 2. give nice errors on failure
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16930) Update the wrapper for fuse-dfs
[
https://issues.apache.org/jira/browse/HDFS-16930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17695655#comment-17695655
]
ASF GitHub Bot commented on HDFS-16930:
---
hadoop-yetus commented on PR #5449:
URL: https://github.com/apache/hadoop/pull/5449#issuecomment-1451675091
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 37m 41s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5449/1/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | mvnsite | 0m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 31s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 17s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | shadedclient | 19m 1s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 3m 38s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 97m 29s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5449/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5449 |
| Optional Tests | dupname asflicense mvnsite unit codespell detsecrets
shellcheck shelldocs |
| uname | Linux a0da5c06db93 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 685625159e24efbec4fb45f10a0cb6602b848337 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5449/1/testReport/ |
| Max. process+thread count | 558 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5449/1/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Update the wrapper for fuse-dfs
> ---
>
> Key: HDFS-16930
> URL: https://issues.apache.org/jira/browse/HDFS-16930
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fuse-dfs
>Reporter: Chao-Heng Lee
>Assignee: Chao-Heng Lee
>Priority: Minor
> Labels: pull-request-available
>
> The fuse_dfs_wrapper.sh hasn't been updated for quite a long time.
> Although the documentation mentions that the script may not work out of the
> box, it would be clearer to update any outdated paths.
> for example, from
> {code:java}
> export
> LIBHDFS_PATH="$HADOOP_HOME/hadoop-hdfs-project/hadoop-hdfs-native-client/target/usr/local/lib"{code}
> to
> {code:java}
> export
> LIBHDFS_PATH="$HADOOP_HOME/hadoop-hdfs-project/hadoop-hdfs-native-client/target/native/target/usr/local/lib"{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16930) Update the wrapper for fuse-dfs
[
https://issues.apache.org/jira/browse/HDFS-16930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-16930:
--
Labels: pull-request-available (was: )
> Update the wrapper for fuse-dfs
> ---
>
> Key: HDFS-16930
> URL: https://issues.apache.org/jira/browse/HDFS-16930
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fuse-dfs
>Reporter: Chao-Heng Lee
>Assignee: Chao-Heng Lee
>Priority: Minor
> Labels: pull-request-available
>
> The fuse_dfs_wrapper.sh hasn't been updated for quite a long time.
> Although the documentation mentions that the script may not work out of the
> box, it would be clearer to update any outdated paths.
> for example, from
> {code:java}
> export
> LIBHDFS_PATH="$HADOOP_HOME/hadoop-hdfs-project/hadoop-hdfs-native-client/target/usr/local/lib"{code}
> to
> {code:java}
> export
> LIBHDFS_PATH="$HADOOP_HOME/hadoop-hdfs-project/hadoop-hdfs-native-client/target/native/target/usr/local/lib"{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16930) Update the wrapper for fuse-dfs
[
https://issues.apache.org/jira/browse/HDFS-16930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17695575#comment-17695575
]
ASF GitHub Bot commented on HDFS-16930:
---
chaohengstudent opened a new pull request, #5449:
URL: https://github.com/apache/hadoop/pull/5449
### Description of PR
update path with LIBHDFS_PATH
add path to LD_LIBRARY_PATH with compatible to java11
edit CLASSPATH
### How was this patch tested?
manual test
```
root@6b76602de66c:/opt/hadoop# ./fuse_dfs_wrapper.sh -d
hdfs://192.168.103.44:14370 /mnt/hdfs
INFO
/home/chaoheng/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/fuse_options.c:115
Ignoring option -d
INFO
/home/chaoheng/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/fuse_options.c:164
Adding FUSE arg /mnt/hdfs
FUSE library version: 2.9.9
nullpath_ok: 0
nopath: 0
utime_omit_ok: 0
unique: 2, opcode: INIT (26), nodeid: 0, insize: 104, pid: 0
INIT: 7.36
flags=0x73fb
max_readahead=0x0002
INFO
/home/chaoheng/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/fuse_init.c:98
Mounting with options: [ protected=(NULL), nn_uri=hdfs://192.168.103.44:14370,
nn_port=0, debug=0, read_only=0, initchecks=0, no_permissions=0, usetrash=0,
entry_timeout=60, attribute_timeout=60, rdbuffer_size=10485760, direct_io=0 ]
fuseConnectInit: initialized with timer period 5, expiry period 300
INIT: 7.19
flags=0x0039
max_readahead=0x0002
max_write=0x0002
max_background=0
congestion_threshold=0
unique: 2, success, outsize: 40
```
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> Update the wrapper for fuse-dfs
> ---
>
> Key: HDFS-16930
> URL: https://issues.apache.org/jira/browse/HDFS-16930
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fuse-dfs
>Reporter: Chao-Heng Lee
>Assignee: Chao-Heng Lee
>Priority: Minor
>
> The fuse_dfs_wrapper.sh hasn't been updated for quite a long time.
> Although the documentation mentions that the script may not work out of the
> box, it would be clearer to update any outdated paths.
> for example, from
> {code:java}
> export
> LIBHDFS_PATH="$HADOOP_HOME/hadoop-hdfs-project/hadoop-hdfs-native-client/target/usr/local/lib"{code}
> to
> {code:java}
> export
> LIBHDFS_PATH="$HADOOP_HOME/hadoop-hdfs-project/hadoop-hdfs-native-client/target/native/target/usr/local/lib"{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
