[jira] [Resolved] (HDFS-16258) HDFS-13671 breaks TestBlockManager in branch-3.2
[ https://issues.apache.org/jira/browse/HDFS-16258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka resolved HDFS-16258. -- Resolution: Cannot Reproduce It passed in the latest qbt job. Closing. https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.2-java8-linux-x86_64/15/testReport/org.apache.hadoop.hdfs.server.blockmanagement/TestBlockManager/ Please feel free to reopen this if the test fails in a specific environment. > HDFS-13671 breaks TestBlockManager in branch-3.2 > > > Key: HDFS-16258 > URL: https://issues.apache.org/jira/browse/HDFS-16258 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.2.3 >Reporter: Wei-Chiu Chuang >Priority: Blocker > > TestBlockManager in branch-3.2 has two failed tests: > * testDeleteCorruptReplicaWithStatleStorages > * testBlockManagerMachinesArray > Looks like broken by HDFS-13671. CC: [~brahmareddy] > Branch-3.3 seems fine. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16262) Async refresh of cached locations in DFSInputStream
[
https://issues.apache.org/jira/browse/HDFS-16262?focusedWorklogId=661307=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-661307
]
ASF GitHub Bot logged work on HDFS-16262:
-
Author: ASF GitHub Bot
Created on: 07/Oct/21 04:35
Start Date: 07/Oct/21 04:35
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3527:
URL: https://github.com/apache/hadoop/pull/3527#issuecomment-937438221
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 51s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 57s | | trunk passed |
| +1 :green_heart: | compile | 4m 56s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 4m 38s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 15s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 40s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 6s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 5m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 3s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 0s | | the patch passed |
| +1 :green_heart: | compile | 5m 2s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 5m 2s | | the patch passed |
| +1 :green_heart: | compile | 4m 37s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 4m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 9s |
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3527/1/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
| hadoop-hdfs-project: The patch generated 15 new + 105 unchanged - 0 fixed =
120 total (was 105) |
| +1 :green_heart: | mvnsite | 2m 4s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 23s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 53s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 5m 39s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 59s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 22s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 231m 45s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3527/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 356m 40s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.tools.TestHdfsConfigFields |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3527/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3527 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 3b643776302b 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19
23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk /
[jira] [Work logged] (HDFS-16257) [HDFS] [RBF] Guava cache performance issue in Router MountTableResolver
[ https://issues.apache.org/jira/browse/HDFS-16257?focusedWorklogId=661297=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-661297 ] ASF GitHub Bot logged work on HDFS-16257: - Author: ASF GitHub Bot Created on: 07/Oct/21 04:05 Start Date: 07/Oct/21 04:05 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #3524: URL: https://github.com/apache/hadoop/pull/3524#issuecomment-937425783 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 9m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ branch-2.10 Compile Tests _ | | +1 :green_heart: | mvninstall | 15m 25s | branch-2.10 passed | | +1 :green_heart: | compile | 0m 38s | branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | +1 :green_heart: | compile | 0m 29s | branch-2.10 passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 | | +1 :green_heart: | checkstyle | 0m 20s | branch-2.10 passed | | +1 :green_heart: | mvnsite | 0m 37s | branch-2.10 passed | | +1 :green_heart: | javadoc | 0m 50s | branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | +1 :green_heart: | javadoc | 0m 33s | branch-2.10 passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 | | +0 :ok: | spotbugs | 3m 36s | Both FindBugs and SpotBugs are enabled, using SpotBugs. | | +1 :green_heart: | spotbugs | 1m 9s | branch-2.10 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 29s | the patch passed | | +1 :green_heart: | compile | 0m 29s | the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | +1 :green_heart: | javac | 0m 29s | the patch passed | | +1 :green_heart: | compile | 0m 25s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 | | +1 :green_heart: | javac | 0m 25s | the patch passed | | +1 :green_heart: | checkstyle | 0m 14s | the patch passed | | +1 :green_heart: | mvnsite | 0m 30s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | +1 :green_heart: | javadoc | 0m 30s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 | | +1 :green_heart: | spotbugs | 1m 10s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 17m 10s | hadoop-hdfs-rbf in the patch passed. | | +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. | | | | 55m 42s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3524/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/3524 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle | | uname | Linux bca894c0206c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | branch-2.10 / dc03afc | | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 | | Multi-JDK versions | /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_262-b10 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3524/4/testReport/ | | Max. process+thread count | 1228 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3524/4/console | | versions | git=2.7.4 maven=3.3.9 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To
[jira] [Resolved] (HDFS-15018) DataNode doesn't shutdown although the number of failed disks reaches dfs.datanode.failed.volumes.tolerated
[
https://issues.apache.org/jira/browse/HDFS-15018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma resolved HDFS-15018.
-
Resolution: Duplicate
> DataNode doesn't shutdown although the number of failed disks reaches
> dfs.datanode.failed.volumes.tolerated
> ---
>
> Key: HDFS-15018
> URL: https://issues.apache.org/jira/browse/HDFS-15018
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.3
> Environment: HDP-2.6.5
>Reporter: Toshihiro Suzuki
>Priority: Major
> Attachments: thread_dumps.txt
>
>
> In our case, we set dfs.datanode.failed.volumes.tolerated=0 but a DataNode
> didn't shutdown when a disk in the DataNode host got failed for some reason.
> The the following log messages were shown in the DataNode log which indicates
> the DataNode detected the disk failure, but the DataNode didn't shutdown:
> {code}
> 2019-09-17T13:15:43.262-0400 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskErrorAsync callback
> got 1 failed volumes: [/data2/hdfs/current]
> 2019-09-17T13:15:43.262-0400 INFO
> org.apache.hadoop.hdfs.server.datanode.BlockScanner: Removing scanner for
> volume /data2/hdfs (StorageID DS-329dec9d-a476-4334-9570-651a7e4d1f44)
> 2019-09-17T13:15:43.263-0400 INFO
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner:
> VolumeScanner(/data2/hdfs, DS-329dec9d-a476-4334-9570-651a7e4d1f44) exiting.
> {code}
> Looking at the HDFS code, it looks like when the DataNode detects a disk
> failure, DataNode waits until the volume reference of the disk is released.
> https://github.com/hortonworks/hadoop/blob/HDP-2.6.5.0-292-tag/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java#L246
> I'm suspecting that the volume reference is not released after the failure
> detection, but not sure the reason.
> And we took thread dumps when the issue was happening. It looks like the
> following thread is waiting for the volume reference of the disk to be
> released:
> {code}
> "pool-4-thread-1" #174 daemon prio=5 os_prio=0 tid=0x7f9e7c7bf800
> nid=0x8325 in Object.wait() [0x7f9e629cb000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList.waitVolumeRemoved(FsVolumeList.java:262)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList.handleVolumeFailures(FsVolumeList.java:246)
> - locked <0x000670559278> (a java.lang.Object)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.handleVolumeFailures(FsDatasetImpl.java:2178)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.handleVolumeFailures(DataNode.java:3410)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.access$100(DataNode.java:248)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode$4.call(DataNode.java:2013)
> at
> org.apache.hadoop.hdfs.server.datanode.checker.DatasetVolumeChecker$ResultHandler.invokeCallback(DatasetVolumeChecker.java:394)
> at
> org.apache.hadoop.hdfs.server.datanode.checker.DatasetVolumeChecker$ResultHandler.cleanup(DatasetVolumeChecker.java:387)
> at
> org.apache.hadoop.hdfs.server.datanode.checker.DatasetVolumeChecker$ResultHandler.onFailure(DatasetVolumeChecker.java:370)
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:977)
> at
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
> at
> org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.executeListener(AbstractFuture.java:991)
> at
> org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.complete(AbstractFuture.java:885)
> at
> org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.setException(AbstractFuture.java:739)
> at
> org.apache.hadoop.hdfs.server.datanode.checker.TimeoutFuture$Fire.run(TimeoutFuture.java:137)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at
[jira] [Commented] (HDFS-15018) DataNode doesn't shutdown although the number of failed disks reaches dfs.datanode.failed.volumes.tolerated
[
https://issues.apache.org/jira/browse/HDFS-15018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425328#comment-17425328
]
Takanobu Asanuma commented on HDFS-15018:
-
We had the same problem. After investigating it, we believe this is fixed by
HDFS-15963.
I'd like to close this jira. If the problem still exist, please reopen it.
> DataNode doesn't shutdown although the number of failed disks reaches
> dfs.datanode.failed.volumes.tolerated
> ---
>
> Key: HDFS-15018
> URL: https://issues.apache.org/jira/browse/HDFS-15018
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.3
> Environment: HDP-2.6.5
>Reporter: Toshihiro Suzuki
>Priority: Major
> Attachments: thread_dumps.txt
>
>
> In our case, we set dfs.datanode.failed.volumes.tolerated=0 but a DataNode
> didn't shutdown when a disk in the DataNode host got failed for some reason.
> The the following log messages were shown in the DataNode log which indicates
> the DataNode detected the disk failure, but the DataNode didn't shutdown:
> {code}
> 2019-09-17T13:15:43.262-0400 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskErrorAsync callback
> got 1 failed volumes: [/data2/hdfs/current]
> 2019-09-17T13:15:43.262-0400 INFO
> org.apache.hadoop.hdfs.server.datanode.BlockScanner: Removing scanner for
> volume /data2/hdfs (StorageID DS-329dec9d-a476-4334-9570-651a7e4d1f44)
> 2019-09-17T13:15:43.263-0400 INFO
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner:
> VolumeScanner(/data2/hdfs, DS-329dec9d-a476-4334-9570-651a7e4d1f44) exiting.
> {code}
> Looking at the HDFS code, it looks like when the DataNode detects a disk
> failure, DataNode waits until the volume reference of the disk is released.
> https://github.com/hortonworks/hadoop/blob/HDP-2.6.5.0-292-tag/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java#L246
> I'm suspecting that the volume reference is not released after the failure
> detection, but not sure the reason.
> And we took thread dumps when the issue was happening. It looks like the
> following thread is waiting for the volume reference of the disk to be
> released:
> {code}
> "pool-4-thread-1" #174 daemon prio=5 os_prio=0 tid=0x7f9e7c7bf800
> nid=0x8325 in Object.wait() [0x7f9e629cb000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList.waitVolumeRemoved(FsVolumeList.java:262)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList.handleVolumeFailures(FsVolumeList.java:246)
> - locked <0x000670559278> (a java.lang.Object)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.handleVolumeFailures(FsDatasetImpl.java:2178)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.handleVolumeFailures(DataNode.java:3410)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.access$100(DataNode.java:248)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode$4.call(DataNode.java:2013)
> at
> org.apache.hadoop.hdfs.server.datanode.checker.DatasetVolumeChecker$ResultHandler.invokeCallback(DatasetVolumeChecker.java:394)
> at
> org.apache.hadoop.hdfs.server.datanode.checker.DatasetVolumeChecker$ResultHandler.cleanup(DatasetVolumeChecker.java:387)
> at
> org.apache.hadoop.hdfs.server.datanode.checker.DatasetVolumeChecker$ResultHandler.onFailure(DatasetVolumeChecker.java:370)
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:977)
> at
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
> at
> org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.executeListener(AbstractFuture.java:991)
> at
> org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.complete(AbstractFuture.java:885)
> at
> org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.setException(AbstractFuture.java:739)
> at
> org.apache.hadoop.hdfs.server.datanode.checker.TimeoutFuture$Fire.run(TimeoutFuture.java:137)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
>
[jira] [Work logged] (HDFS-16257) [HDFS] [RBF] Guava cache performance issue in Router MountTableResolver
[ https://issues.apache.org/jira/browse/HDFS-16257?focusedWorklogId=661293=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-661293 ] ASF GitHub Bot logged work on HDFS-16257: - Author: ASF GitHub Bot Created on: 07/Oct/21 03:22 Start Date: 07/Oct/21 03:22 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #3524: URL: https://github.com/apache/hadoop/pull/3524#issuecomment-937412397 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ branch-2.10 Compile Tests _ | | +1 :green_heart: | mvninstall | 15m 16s | branch-2.10 passed | | +1 :green_heart: | compile | 0m 33s | branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | +1 :green_heart: | compile | 0m 27s | branch-2.10 passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 | | +1 :green_heart: | checkstyle | 0m 20s | branch-2.10 passed | | +1 :green_heart: | mvnsite | 0m 39s | branch-2.10 passed | | +1 :green_heart: | javadoc | 0m 49s | branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | +1 :green_heart: | javadoc | 0m 33s | branch-2.10 passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 | | +0 :ok: | spotbugs | 3m 41s | Both FindBugs and SpotBugs are enabled, using SpotBugs. | | +1 :green_heart: | spotbugs | 1m 12s | branch-2.10 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 31s | the patch passed | | +1 :green_heart: | compile | 0m 31s | the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | +1 :green_heart: | javac | 0m 31s | the patch passed | | +1 :green_heart: | compile | 0m 24s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 | | +1 :green_heart: | javac | 0m 24s | the patch passed | | +1 :green_heart: | checkstyle | 0m 15s | the patch passed | | +1 :green_heart: | mvnsite | 0m 28s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | javadoc | 0m 44s | the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | +1 :green_heart: | javadoc | 0m 31s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 | | +1 :green_heart: | spotbugs | 1m 11s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 17m 15s | hadoop-hdfs-rbf in the patch passed. | | +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. | | | | 47m 4s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3524/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/3524 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle | | uname | Linux 3d82906a0f67 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | branch-2.10 / dc03afc | | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 | | Multi-JDK versions | /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_262-b10 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3524/3/testReport/ | | Max. process+thread count | 1445 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3524/3/console | | versions | git=2.7.4 maven=3.3.9 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To
[jira] [Commented] (HDFS-16259) Catch and re-throw sub-classes of AccessControlException thrown by any permission provider plugins (eg Ranger)
[
https://issues.apache.org/jira/browse/HDFS-16259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425324#comment-17425324
]
Ayush Saxena commented on HDFS-16259:
-
Isn't that Ranger problem? It should return an exception which can be decoded.
I see the component marked as Namenode, Do you plan to add such a logic in the
Namenode side, that it catches any sub-class ACE & convert to ACE. This doesn't
look like something we should do, Whatever the plugin returns we should
propagate that back, Behaviour for that plugin should be same for all services,
not something for the Namenode.
What special Ranger is trying to do with RangerAccessControlException, that
they can't change it to ACE?
Not very sure about the use case, but is there a problem due to unwrapping did
you try handling in the DFSClient? Something like:
{noformat}
diff --git
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
index 58906846298..09c68a932af 100755
---
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
+++
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
@@ -2960,9 +2960,7 @@ public void checkAccess(String src, FsAction mode) throws
IOException {
try (TraceScope ignored = newPathTraceScope("checkAccess", src)) {
namenode.checkAccess(src, mode);
} catch (RemoteException re) {
- throw re.unwrapRemoteException(AccessControlException.class,
- FileNotFoundException.class,
- UnresolvedPathException.class);
+ throw re.unwrapRemoteException();
}
}
{noformat}
> Catch and re-throw sub-classes of AccessControlException thrown by any
> permission provider plugins (eg Ranger)
> --
>
> Key: HDFS-16259
> URL: https://issues.apache.org/jira/browse/HDFS-16259
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
>
> When a permission provider plugin is enabled (eg Ranger) there are some
> scenarios where it can throw a sub-class of an AccessControlException (eg
> RangerAccessControlException). If this exception is allowed to propagate up
> the stack, it can give problems in the HDFS Client, when it unwraps the
> remote exception containing the AccessControlException sub-class.
> Ideally, we should make AccessControlException final so it cannot be
> sub-classed, but that would be a breaking change at this point. Therefore I
> believe the safest thing to do, is to catch any AccessControlException that
> comes out of the permission enforcer plugin, and re-throw an
> AccessControlException instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16257) [HDFS] [RBF] Guava cache performance issue in Router MountTableResolver
[ https://issues.apache.org/jira/browse/HDFS-16257?focusedWorklogId=661290=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-661290 ] ASF GitHub Bot logged work on HDFS-16257: - Author: ASF GitHub Bot Created on: 07/Oct/21 03:15 Start Date: 07/Oct/21 03:15 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #3524: URL: https://github.com/apache/hadoop/pull/3524#issuecomment-937409811 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 9m 33s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ branch-2.10 Compile Tests _ | | +1 :green_heart: | mvninstall | 15m 11s | branch-2.10 passed | | +1 :green_heart: | compile | 0m 35s | branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | +1 :green_heart: | compile | 0m 28s | branch-2.10 passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 | | +1 :green_heart: | checkstyle | 0m 21s | branch-2.10 passed | | +1 :green_heart: | mvnsite | 0m 37s | branch-2.10 passed | | +1 :green_heart: | javadoc | 0m 49s | branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | +1 :green_heart: | javadoc | 0m 34s | branch-2.10 passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 | | +0 :ok: | spotbugs | 3m 37s | Both FindBugs and SpotBugs are enabled, using SpotBugs. | | +1 :green_heart: | spotbugs | 1m 7s | branch-2.10 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 28s | the patch passed | | +1 :green_heart: | compile | 0m 31s | the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | +1 :green_heart: | javac | 0m 31s | the patch passed | | +1 :green_heart: | compile | 0m 24s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 | | +1 :green_heart: | javac | 0m 24s | the patch passed | | +1 :green_heart: | checkstyle | 0m 14s | the patch passed | | +1 :green_heart: | mvnsite | 0m 28s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | +1 :green_heart: | javadoc | 0m 29s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 | | +1 :green_heart: | spotbugs | 1m 11s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 17m 10s | hadoop-hdfs-rbf in the patch passed. | | +1 :green_heart: | asflicense | 0m 50s | The patch does not generate ASF License warnings. | | | | 56m 0s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3524/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/3524 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle | | uname | Linux 356e4376ce9f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | branch-2.10 / dc03afc | | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 | | Multi-JDK versions | /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_262-b10 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3524/2/testReport/ | | Max. process+thread count | 1092 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3524/2/console | | versions | git=2.7.4 maven=3.3.9 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To
[jira] [Work logged] (HDFS-16257) [HDFS] [RBF] Guava cache performance issue in Router MountTableResolver
[ https://issues.apache.org/jira/browse/HDFS-16257?focusedWorklogId=661279=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-661279 ] ASF GitHub Bot logged work on HDFS-16257: - Author: ASF GitHub Bot Created on: 07/Oct/21 02:19 Start Date: 07/Oct/21 02:19 Worklog Time Spent: 10m Work Description: symious commented on a change in pull request #3524: URL: https://github.com/apache/hadoop/pull/3524#discussion_r72379 ## File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java ## @@ -138,6 +138,8 @@ public MountTableResolver(Configuration conf, Router routerService, FEDERATION_MOUNT_TABLE_MAX_CACHE_SIZE, FEDERATION_MOUNT_TABLE_MAX_CACHE_SIZE_DEFAULT); this.locationCache = CacheBuilder.newBuilder() + // To warkaround guava bug https://github.com/google/guava/issues/1055 Review comment: Updated, please help to check. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 661279) Time Spent: 1h 10m (was: 1h) > [HDFS] [RBF] Guava cache performance issue in Router MountTableResolver > --- > > Key: HDFS-16257 > URL: https://issues.apache.org/jira/browse/HDFS-16257 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.10.1 >Reporter: Janus Chow >Assignee: Janus Chow >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > Branch 2.10.1 uses guava version of 11.0.2, which has a bug which affects the > performance of cache, which was mentioned in HDFS-13821. > Since upgrading guava version seems affecting too much, this ticket is to add > a configuration setting when initializing cache to walk around this issue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16258) HDFS-13671 breaks TestBlockManager in branch-3.2
[ https://issues.apache.org/jira/browse/HDFS-16258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425311#comment-17425311 ] Akira Ajisaka commented on HDFS-16258: -- I couldn't reproduce the failure in branch-3.2. The commit hash is 51d17f3cb399ae0ed66d3a2357ef091151d25646. > HDFS-13671 breaks TestBlockManager in branch-3.2 > > > Key: HDFS-16258 > URL: https://issues.apache.org/jira/browse/HDFS-16258 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.2.3 >Reporter: Wei-Chiu Chuang >Priority: Blocker > > TestBlockManager in branch-3.2 has two failed tests: > * testDeleteCorruptReplicaWithStatleStorages > * testBlockManagerMachinesArray > Looks like broken by HDFS-13671. CC: [~brahmareddy] > Branch-3.3 seems fine. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16262) Async refresh of cached locations in DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-16262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16262: -- Labels: pull-request-available (was: ) > Async refresh of cached locations in DFSInputStream > --- > > Key: HDFS-16262 > URL: https://issues.apache.org/jira/browse/HDFS-16262 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > HDFS-15119 added the ability to invalidate cached block locations in > DFSInputStream. As written, the feature will affect all DFSInputStreams > regardless of whether they need it or not. The invalidation also only applies > on the next request, so the next request will pay the cost of calling > openInfo before reading the data. > I'm working on a feature for HBase which enables efficient healing of > locality through Balancer-style low level block moves (HBASE-26250). I'd like > to utilize the idea started in HDFS-15119 in order to update DFSInputStreams > after blocks have been moved to local hosts. > I was considering using the feature as is, but some of our clusters are quite > large and I'm concerned about the impact on the namenode: > * We have some clusters with over 350k StoreFiles, so that'd be 350k > DFSInputStreams. With such a large number and very active usage, having the > refresh be in-line makes it too hard to ensure we don't DDOS the NameNode. > * Currently we need to pay the price of openInfo the next time a > DFSInputStream is invoked. Moving that async would minimize the latency hit. > Also, some StoreFiles might be far less frequently accessed, so they may live > on for a long time before ever refreshing. We'd like to be able to know that > all DFSInputStreams are refreshed by a given time. > * We may have 350k files, but only a small percentage of them are ever > non-local at a given time. Refreshing only if necessary will save a lot of > work. > In order to make this as painless to end users as possible, I'd like to: > * Update the implementation to utilize an async thread for managing > refreshes. This will give more control over rate limiting across all > DFSInputStreams in a DFSClient, and also ensure that all DFSInputStreams are > refreshed. > * Only refresh files which are lacking a local replica or have known > deadNodes to be cleaned up > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16262) Async refresh of cached locations in DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-16262?focusedWorklogId=661230=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-661230 ] ASF GitHub Bot logged work on HDFS-16262: - Author: ASF GitHub Bot Created on: 06/Oct/21 22:37 Start Date: 06/Oct/21 22:37 Worklog Time Spent: 10m Work Description: bbeaudreault opened a new pull request #3527: URL: https://github.com/apache/hadoop/pull/3527 ### Description of PR Refactor refreshing of cached block locations so that it happens as part of an async process, with rate limiting. Add the ability to limit to only refresh DFSInputStreams if necessary. This defaults to false to preserve backwards compatibility with the old behavior from https://issues.apache.org/jira/browse/HDFS-15119 See https://issues.apache.org/jira/browse/HDFS-16262 ### How was this patch tested? I added a new test class TestLocatedBlocksRefresher. I am in the process of deploying this internally on one of our hadoop-3.3 clusters, will report back. ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 661230) Remaining Estimate: 0h Time Spent: 10m > Async refresh of cached locations in DFSInputStream > --- > > Key: HDFS-16262 > URL: https://issues.apache.org/jira/browse/HDFS-16262 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > HDFS-15119 added the ability to invalidate cached block locations in > DFSInputStream. As written, the feature will affect all DFSInputStreams > regardless of whether they need it or not. The invalidation also only applies > on the next request, so the next request will pay the cost of calling > openInfo before reading the data. > I'm working on a feature for HBase which enables efficient healing of > locality through Balancer-style low level block moves (HBASE-26250). I'd like > to utilize the idea started in HDFS-15119 in order to update DFSInputStreams > after blocks have been moved to local hosts. > I was considering using the feature as is, but some of our clusters are quite > large and I'm concerned about the impact on the namenode: > * We have some clusters with over 350k StoreFiles, so that'd be 350k > DFSInputStreams. With such a large number and very active usage, having the > refresh be in-line makes it too hard to ensure we don't DDOS the NameNode. > * Currently we need to pay the price of openInfo the next time a > DFSInputStream is invoked. Moving that async would minimize the latency hit. > Also, some StoreFiles might be far less frequently accessed, so they may live > on for a long time before ever refreshing. We'd like to be able to know that > all DFSInputStreams are refreshed by a given time. > * We may have 350k files, but only a small percentage of them are ever > non-local at a given time. Refreshing only if necessary will save a lot of > work. > In order to make this as painless to end users as possible, I'd like to: > * Update the implementation to utilize an async thread for managing > refreshes. This will give more control over rate limiting across all > DFSInputStreams in a DFSClient, and also ensure that all DFSInputStreams are > refreshed. > * Only refresh files which are lacking a local replica or have known > deadNodes to be cleaned up > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16262) Async refresh of cached locations in DFSInputStream
Bryan Beaudreault created HDFS-16262: Summary: Async refresh of cached locations in DFSInputStream Key: HDFS-16262 URL: https://issues.apache.org/jira/browse/HDFS-16262 Project: Hadoop HDFS Issue Type: Improvement Reporter: Bryan Beaudreault Assignee: Bryan Beaudreault HDFS-15119 added the ability to invalidate cached block locations in DFSInputStream. As written, the feature will affect all DFSInputStreams regardless of whether they need it or not. The invalidation also only applies on the next request, so the next request will pay the cost of calling openInfo before reading the data. I'm working on a feature for HBase which enables efficient healing of locality through Balancer-style low level block moves (HBASE-26250). I'd like to utilize the idea started in HDFS-15119 in order to update DFSInputStreams after blocks have been moved to local hosts. I was considering using the feature as is, but some of our clusters are quite large and I'm concerned about the impact on the namenode: * We have some clusters with over 350k StoreFiles, so that'd be 350k DFSInputStreams. With such a large number and very active usage, having the refresh be in-line makes it too hard to ensure we don't DDOS the NameNode. * Currently we need to pay the price of openInfo the next time a DFSInputStream is invoked. Moving that async would minimize the latency hit. Also, some StoreFiles might be far less frequently accessed, so they may live on for a long time before ever refreshing. We'd like to be able to know that all DFSInputStreams are refreshed by a given time. * We may have 350k files, but only a small percentage of them are ever non-local at a given time. Refreshing only if necessary will save a lot of work. In order to make this as painless to end users as possible, I'd like to: * Update the implementation to utilize an async thread for managing refreshes. This will give more control over rate limiting across all DFSInputStreams in a DFSClient, and also ensure that all DFSInputStreams are refreshed. * Only refresh files which are lacking a local replica or have known deadNodes to be cleaned up -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16261) Configurable grace period around deletion of invalidated blocks
[ https://issues.apache.org/jira/browse/HDFS-16261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HDFS-16261: - Description: When a block is moved with REPLACE_BLOCK, the new location is recorded in the NameNode and the NameNode instructs the old host to in invalidate the block using DNA_INVALIDATE. As it stands today, this invalidation is async but tends to happen relatively quickly. I'm working on a feature for HBase which enables efficient healing of locality through Balancer-style low level block moves (HBASE-26250). One issue is that HBase tends to keep open long running DFSInputStreams and moving blocks from under them causes lots of warns in the RegionServer and increases long tail latencies due to the necessary retries in the DFSClient. One way I'd like to fix this is to provide a configurable grace period on async invalidations. This would give the DFSClient enough time to refresh block locations before hitting any errors. was: When a block is moved with REPLACE_BLOCK, the new location is recorded in the NameNode and the NameNode instructs the old host to in invalidate the block using DNA_INVALIDATE. As it stands today, this invalidation is async but tends to happen relatively quickly. I'm working on a feature for HBase which enables efficient healing of locality through Balancer-style low level block moves. One issue is that HBase tends to keep open long running DFSInputStreams and moving blocks from under them causes lots of warns in the RegionServer and increases long tail latencies due to the necessary retries in the DFSClient. One way I'd like to fix this is to provide a configurable grace period on async invalidations. This would give the DFSClient enough time to refresh block locations before hitting any errors. > Configurable grace period around deletion of invalidated blocks > --- > > Key: HDFS-16261 > URL: https://issues.apache.org/jira/browse/HDFS-16261 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Major > > When a block is moved with REPLACE_BLOCK, the new location is recorded in the > NameNode and the NameNode instructs the old host to in invalidate the block > using DNA_INVALIDATE. As it stands today, this invalidation is async but > tends to happen relatively quickly. > I'm working on a feature for HBase which enables efficient healing of > locality through Balancer-style low level block moves (HBASE-26250). One > issue is that HBase tends to keep open long running DFSInputStreams and > moving blocks from under them causes lots of warns in the RegionServer and > increases long tail latencies due to the necessary retries in the DFSClient. > One way I'd like to fix this is to provide a configurable grace period on > async invalidations. This would give the DFSClient enough time to refresh > block locations before hitting any errors. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16261) Configurable grace period around deletion of invalidated blocks
Bryan Beaudreault created HDFS-16261: Summary: Configurable grace period around deletion of invalidated blocks Key: HDFS-16261 URL: https://issues.apache.org/jira/browse/HDFS-16261 Project: Hadoop HDFS Issue Type: New Feature Reporter: Bryan Beaudreault Assignee: Bryan Beaudreault When a block is moved with REPLACE_BLOCK, the new location is recorded in the NameNode and the NameNode instructs the old host to in invalidate the block using DNA_INVALIDATE. As it stands today, this invalidation is async but tends to happen relatively quickly. I'm working on a feature for HBase which enables efficient healing of locality through Balancer-style low level block moves. One issue is that HBase tends to keep open long running DFSInputStreams and moving blocks from under them causes lots of warns in the RegionServer and increases long tail latencies due to the necessary retries in the DFSClient. One way I'd like to fix this is to provide a configurable grace period on async invalidations. This would give the DFSClient enough time to refresh block locations before hitting any errors. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16257) [HDFS] [RBF] Guava cache performance issue in Router MountTableResolver
[ https://issues.apache.org/jira/browse/HDFS-16257?focusedWorklogId=661139=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-661139 ] ASF GitHub Bot logged work on HDFS-16257: - Author: ASF GitHub Bot Created on: 06/Oct/21 19:09 Start Date: 06/Oct/21 19:09 Worklog Time Spent: 10m Work Description: goiri commented on a change in pull request #3524: URL: https://github.com/apache/hadoop/pull/3524#discussion_r723600325 ## File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java ## @@ -138,6 +138,8 @@ public MountTableResolver(Configuration conf, Router routerService, FEDERATION_MOUNT_TABLE_MAX_CACHE_SIZE, FEDERATION_MOUNT_TABLE_MAX_CACHE_SIZE_DEFAULT); this.locationCache = CacheBuilder.newBuilder() + // To warkaround guava bug https://github.com/google/guava/issues/1055 Review comment: mention in what version this is fixed ## File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java ## @@ -138,6 +138,8 @@ public MountTableResolver(Configuration conf, Router routerService, FEDERATION_MOUNT_TABLE_MAX_CACHE_SIZE, FEDERATION_MOUNT_TABLE_MAX_CACHE_SIZE_DEFAULT); this.locationCache = CacheBuilder.newBuilder() + // To warkaround guava bug https://github.com/google/guava/issues/1055 Review comment: work around -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 661139) Time Spent: 1h (was: 50m) > [HDFS] [RBF] Guava cache performance issue in Router MountTableResolver > --- > > Key: HDFS-16257 > URL: https://issues.apache.org/jira/browse/HDFS-16257 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.10.1 >Reporter: Janus Chow >Assignee: Janus Chow >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Branch 2.10.1 uses guava version of 11.0.2, which has a bug which affects the > performance of cache, which was mentioned in HDFS-13821. > Since upgrading guava version seems affecting too much, this ticket is to add > a configuration setting when initializing cache to walk around this issue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16260) Make hdfs_deleteSnapshot tool cross platform
Gautham Banasandra created HDFS-16260: - Summary: Make hdfs_deleteSnapshot tool cross platform Key: HDFS-16260 URL: https://issues.apache.org/jira/browse/HDFS-16260 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client, libhdfs++, tools Affects Versions: 3.4.0 Environment: Centos 7, Centos 8, Debian 10, Ubuntu Focal Reporter: Gautham Banasandra Assignee: Gautham Banasandra The source files for hdfs_deleteSnapshot uses *getopt* for parsing the command line arguments. getopt is available only on Linux and thus, isn't cross platform. We need to replace getopt with *boost::program_options* to make this cross platform. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14729) Upgrade Bootstrap and jQuery versions used in HDFS UIs
[
https://issues.apache.org/jira/browse/HDFS-14729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425094#comment-17425094
]
Brahma Reddy Battula commented on HDFS-14729:
-
{quote}Can we backport this to branch-3.2?
{quote}
Sure.. can we raise one Jira for backport..? CC. [~vivekratnavel] and [~sunilg]
> Upgrade Bootstrap and jQuery versions used in HDFS UIs
> --
>
> Key: HDFS-14729
> URL: https://issues.apache.org/jira/browse/HDFS-14729
> Project: Hadoop HDFS
> Issue Type: Task
> Components: ui
>Reporter: Vivek Ratnavel Subramanian
>Assignee: Vivek Ratnavel Subramanian
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14729.v1.patch
>
>
> The current versions of bootstrap and jquery have multiple medium severity
> CVEs reported till date and needs to be updated to the latest versions with
> no reported CVEs.
>
> I suggest updating the following libraries:
> ||Library||From version||To version||
> |Bootstrap|3.3.7|3.4.1|
> |jQuery|3.3.1|3.4.1|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16252) Correct docs for dfs.http.client.retry.policy.spec
[
https://issues.apache.org/jira/browse/HDFS-16252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell updated HDFS-16252:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Correct docs for dfs.http.client.retry.policy.spec
> ---
>
> Key: HDFS-16252
> URL: https://issues.apache.org/jira/browse/HDFS-16252
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Fix For: 3.4.0, 3.3.2
>
> Attachments: HDFS-16252.001.patch, HDFS-16252.002.patch
>
>
> The hdfs-default doc for dfs.http.client.retry.policy.spec is incorrect, as
> it has the wait time and retries switched around in the descriptio. Also, the
> doc for dfs.client.retry.policy.spec is not present and should be the same as
> for dfs.http.client.retry.policy.spec.
> The code shows the timeout is first and then the number of retries:
> {code}
> String POLICY_SPEC_KEY = PREFIX + "policy.spec";
> String POLICY_SPEC_DEFAULT = "1,6,6,10"; //t1,n1,t2,n2,...
> // In RetryPolicies.java, we can see it gets the timeout as the first in
> the pair
>/**
> * Parse the given string as a MultipleLinearRandomRetry object.
> * The format of the string is "t_1, n_1, t_2, n_2, ...",
> * where t_i and n_i are the i-th pair of sleep time and number of
> retries.
> * Note that the white spaces in the string are ignored.
> *
> * @return the parsed object, or null if the parsing fails.
> */
> public static MultipleLinearRandomRetry parseCommaSeparatedString(String
> s) {
> final String[] elements = s.split(",");
> if (elements.length == 0) {
> LOG.warn("Illegal value: there is no element in \"" + s + "\".");
> return null;
> }
> if (elements.length % 2 != 0) {
> LOG.warn("Illegal value: the number of elements in \"" + s + "\" is "
> + elements.length + " but an even number of elements is
> expected.");
> return null;
> }
> final List pairs
> = new ArrayList();
>
> for(int i = 0; i < elements.length; ) {
> //parse the i-th sleep-time
> final int sleep = parsePositiveInt(elements, i++, s);
> if (sleep == -1) {
> return null; //parse fails
> }
> //parse the i-th number-of-retries
> final int retries = parsePositiveInt(elements, i++, s);
> if (retries == -1) {
> return null; //parse fails
> }
> pairs.add(new RetryPolicies.MultipleLinearRandomRetry.Pair(retries,
> sleep));
> }
> return new RetryPolicies.MultipleLinearRandomRetry(pairs);
> }
> {code}
> This change simply updates the docs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16254) Cleanup protobuf on exit of hdfs_allowSnapshot
[ https://issues.apache.org/jira/browse/HDFS-16254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425073#comment-17425073 ] Íñigo Goiri commented on HDFS-16254: Thanks [~gautham] for the patch. Merged PR 3518 to trunk. > Cleanup protobuf on exit of hdfs_allowSnapshot > -- > > Key: HDFS-16254 > URL: https://issues.apache.org/jira/browse/HDFS-16254 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client, libhdfs++, tools >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Need to move the call google::protobuf::ShutdownProtobufLibrary() to main > method instead of > [AllowSnapshot::HandlePath|https://github.com/apache/hadoop/blob/35a8d48872a13438d4c4199b6ef5b902105e2eb2/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs-allow-snapshot/hdfs-allow-snapshot.cc#L116-L117] > since we want the clean-up tasks to run only when the program exits. > The current implementation doesn't cause any issues since > AllowSnapshot::HandlePath is called only once. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16254) Cleanup protobuf on exit of hdfs_allowSnapshot
[ https://issues.apache.org/jira/browse/HDFS-16254?focusedWorklogId=661042=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-661042 ] ASF GitHub Bot logged work on HDFS-16254: - Author: ASF GitHub Bot Created on: 06/Oct/21 16:24 Start Date: 06/Oct/21 16:24 Worklog Time Spent: 10m Work Description: goiri merged pull request #3518: URL: https://github.com/apache/hadoop/pull/3518 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 661042) Time Spent: 1h 10m (was: 1h) > Cleanup protobuf on exit of hdfs_allowSnapshot > -- > > Key: HDFS-16254 > URL: https://issues.apache.org/jira/browse/HDFS-16254 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client, libhdfs++, tools >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Need to move the call google::protobuf::ShutdownProtobufLibrary() to main > method instead of > [AllowSnapshot::HandlePath|https://github.com/apache/hadoop/blob/35a8d48872a13438d4c4199b6ef5b902105e2eb2/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs-allow-snapshot/hdfs-allow-snapshot.cc#L116-L117] > since we want the clean-up tasks to run only when the program exits. > The current implementation doesn't cause any issues since > AllowSnapshot::HandlePath is called only once. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16254) Cleanup protobuf on exit of hdfs_allowSnapshot
[ https://issues.apache.org/jira/browse/HDFS-16254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16254. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Cleanup protobuf on exit of hdfs_allowSnapshot > -- > > Key: HDFS-16254 > URL: https://issues.apache.org/jira/browse/HDFS-16254 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client, libhdfs++, tools >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Need to move the call google::protobuf::ShutdownProtobufLibrary() to main > method instead of > [AllowSnapshot::HandlePath|https://github.com/apache/hadoop/blob/35a8d48872a13438d4c4199b6ef5b902105e2eb2/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs-allow-snapshot/hdfs-allow-snapshot.cc#L116-L117] > since we want the clean-up tasks to run only when the program exits. > The current implementation doesn't cause any issues since > AllowSnapshot::HandlePath is called only once. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16257) [HDFS] [RBF] Guava cache performance issue in Router MountTableResolver
[ https://issues.apache.org/jira/browse/HDFS-16257?focusedWorklogId=661001=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-661001 ] ASF GitHub Bot logged work on HDFS-16257: - Author: ASF GitHub Bot Created on: 06/Oct/21 15:30 Start Date: 06/Oct/21 15:30 Worklog Time Spent: 10m Work Description: symious commented on pull request #3524: URL: https://github.com/apache/hadoop/pull/3524#issuecomment-936503908 @ayushtkn Thanks for the review. Trunk uses guava from org.apache.hadoop.thirdparty.hadoop-shaded-guava:1.1.1, the guava cache in it should have fixed the bug. Tested with the above dependency, the overhead is eliminated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 661001) Time Spent: 50m (was: 40m) > [HDFS] [RBF] Guava cache performance issue in Router MountTableResolver > --- > > Key: HDFS-16257 > URL: https://issues.apache.org/jira/browse/HDFS-16257 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.10.1 >Reporter: Janus Chow >Assignee: Janus Chow >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Branch 2.10.1 uses guava version of 11.0.2, which has a bug which affects the > performance of cache, which was mentioned in HDFS-13821. > Since upgrading guava version seems affecting too much, this ticket is to add > a configuration setting when initializing cache to walk around this issue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16252) Correct docs for dfs.http.client.retry.policy.spec
[
https://issues.apache.org/jira/browse/HDFS-16252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell updated HDFS-16252:
-
Fix Version/s: 3.3.2
3.4.0
> Correct docs for dfs.http.client.retry.policy.spec
> ---
>
> Key: HDFS-16252
> URL: https://issues.apache.org/jira/browse/HDFS-16252
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Fix For: 3.4.0, 3.3.2
>
> Attachments: HDFS-16252.001.patch, HDFS-16252.002.patch
>
>
> The hdfs-default doc for dfs.http.client.retry.policy.spec is incorrect, as
> it has the wait time and retries switched around in the descriptio. Also, the
> doc for dfs.client.retry.policy.spec is not present and should be the same as
> for dfs.http.client.retry.policy.spec.
> The code shows the timeout is first and then the number of retries:
> {code}
> String POLICY_SPEC_KEY = PREFIX + "policy.spec";
> String POLICY_SPEC_DEFAULT = "1,6,6,10"; //t1,n1,t2,n2,...
> // In RetryPolicies.java, we can see it gets the timeout as the first in
> the pair
>/**
> * Parse the given string as a MultipleLinearRandomRetry object.
> * The format of the string is "t_1, n_1, t_2, n_2, ...",
> * where t_i and n_i are the i-th pair of sleep time and number of
> retries.
> * Note that the white spaces in the string are ignored.
> *
> * @return the parsed object, or null if the parsing fails.
> */
> public static MultipleLinearRandomRetry parseCommaSeparatedString(String
> s) {
> final String[] elements = s.split(",");
> if (elements.length == 0) {
> LOG.warn("Illegal value: there is no element in \"" + s + "\".");
> return null;
> }
> if (elements.length % 2 != 0) {
> LOG.warn("Illegal value: the number of elements in \"" + s + "\" is "
> + elements.length + " but an even number of elements is
> expected.");
> return null;
> }
> final List pairs
> = new ArrayList();
>
> for(int i = 0; i < elements.length; ) {
> //parse the i-th sleep-time
> final int sleep = parsePositiveInt(elements, i++, s);
> if (sleep == -1) {
> return null; //parse fails
> }
> //parse the i-th number-of-retries
> final int retries = parsePositiveInt(elements, i++, s);
> if (retries == -1) {
> return null; //parse fails
> }
> pairs.add(new RetryPolicies.MultipleLinearRandomRetry.Pair(retries,
> sleep));
> }
> return new RetryPolicies.MultipleLinearRandomRetry(pairs);
> }
> {code}
> This change simply updates the docs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16257) [HDFS] [RBF] Guava cache performance issue in Router MountTableResolver
[ https://issues.apache.org/jira/browse/HDFS-16257?focusedWorklogId=660970=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660970 ] ASF GitHub Bot logged work on HDFS-16257: - Author: ASF GitHub Bot Created on: 06/Oct/21 14:50 Start Date: 06/Oct/21 14:50 Worklog Time Spent: 10m Work Description: ayushtkn commented on pull request #3524: URL: https://github.com/apache/hadoop/pull/3524#issuecomment-936419231 Can we have this change merged to trunk as well, and then backport it till 2.10? Any pointers, can this have some adverse affects on trunk version of Guava? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 660970) Time Spent: 40m (was: 0.5h) > [HDFS] [RBF] Guava cache performance issue in Router MountTableResolver > --- > > Key: HDFS-16257 > URL: https://issues.apache.org/jira/browse/HDFS-16257 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.10.1 >Reporter: Janus Chow >Assignee: Janus Chow >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Branch 2.10.1 uses guava version of 11.0.2, which has a bug which affects the > performance of cache, which was mentioned in HDFS-13821. > Since upgrading guava version seems affecting too much, this ticket is to add > a configuration setting when initializing cache to walk around this issue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16239) XAttr#toString doesnt print the attribute value in readable format
[ https://issues.apache.org/jira/browse/HDFS-16239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renukaprasad C resolved HDFS-16239. --- Resolution: Invalid To print, have we considered using XattrCodec APIs. Its not neccessary to print the XAttr. > XAttr#toString doesnt print the attribute value in readable format > -- > > Key: HDFS-16239 > URL: https://issues.apache.org/jira/browse/HDFS-16239 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > org.apache.hadoop.fs.XAttr#toString prints the value of attribute in bytes. > return "XAttr [ns=" + ns + ", name=" + name + ", value=" > + Arrays.toString(value) + "]"; > XAttr [ns=SYSTEM, name=az.expression, value=[82, 69, 80, 91, 50, 93..] > This should be converted to String rather than printing to Array of bytes. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16239) XAttr#toString doesnt print the attribute value in readable format
[ https://issues.apache.org/jira/browse/HDFS-16239?focusedWorklogId=660858=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660858 ] ASF GitHub Bot logged work on HDFS-16239: - Author: ASF GitHub Bot Created on: 06/Oct/21 11:22 Start Date: 06/Oct/21 11:22 Worklog Time Spent: 10m Work Description: prasad-acit closed pull request #3491: URL: https://github.com/apache/hadoop/pull/3491 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 660858) Time Spent: 2h (was: 1h 50m) > XAttr#toString doesnt print the attribute value in readable format > -- > > Key: HDFS-16239 > URL: https://issues.apache.org/jira/browse/HDFS-16239 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > org.apache.hadoop.fs.XAttr#toString prints the value of attribute in bytes. > return "XAttr [ns=" + ns + ", name=" + name + ", value=" > + Arrays.toString(value) + "]"; > XAttr [ns=SYSTEM, name=az.expression, value=[82, 69, 80, 91, 50, 93..] > This should be converted to String rather than printing to Array of bytes. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16251) Make hdfs_cat tool cross platform
[
https://issues.apache.org/jira/browse/HDFS-16251?focusedWorklogId=660834=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660834
]
ASF GitHub Bot logged work on HDFS-16251:
-
Author: ASF GitHub Bot
Created on: 06/Oct/21 10:33
Start Date: 06/Oct/21 10:33
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3523:
URL: https://github.com/apache/hadoop/pull/3523#issuecomment-935934414
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 20m 26s | | trunk passed |
| +1 :green_heart: | compile | 2m 49s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 2m 51s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | mvnsite | 0m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 45m 23s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 2m 39s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | cc | 2m 39s | | the patch passed |
| +1 :green_heart: | golang | 2m 39s | | the patch passed |
| +1 :green_heart: | javac | 2m 39s | | the patch passed |
| +1 :green_heart: | compile | 2m 42s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | cc | 2m 42s | | the patch passed |
| +1 :green_heart: | golang | 2m 42s | | the patch passed |
| +1 :green_heart: | javac | 2m 42s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 32s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 32m 2s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 105m 41s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3523/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3523 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell golang |
| uname | Linux 0b09ba10a7ce 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c57a63fbae71e00a84315643fdccf540f2baf026 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3523/3/testReport/ |
| Max. process+thread count | 717 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3523/3/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 660834)
Time Spent: 1h (was: 50m)
> Make hdfs_cat tool cross platform
>
[jira] [Commented] (HDFS-14729) Upgrade Bootstrap and jQuery versions used in HDFS UIs
[ https://issues.apache.org/jira/browse/HDFS-14729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424898#comment-17424898 ] Ananya Singh commented on HDFS-14729: - Can we backport this to branch-3.2? > Upgrade Bootstrap and jQuery versions used in HDFS UIs > -- > > Key: HDFS-14729 > URL: https://issues.apache.org/jira/browse/HDFS-14729 > Project: Hadoop HDFS > Issue Type: Task > Components: ui >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14729.v1.patch > > > The current versions of bootstrap and jquery have multiple medium severity > CVEs reported till date and needs to be updated to the latest versions with > no reported CVEs. > > I suggest updating the following libraries: > ||Library||From version||To version|| > |Bootstrap|3.3.7|3.4.1| > |jQuery|3.3.1|3.4.1| -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16259) Catch and re-throw sub-classes of AccessControlException thrown by any permission provider plugins (eg Ranger)
Stephen O'Donnell created HDFS-16259: Summary: Catch and re-throw sub-classes of AccessControlException thrown by any permission provider plugins (eg Ranger) Key: HDFS-16259 URL: https://issues.apache.org/jira/browse/HDFS-16259 Project: Hadoop HDFS Issue Type: Bug Components: namenode Reporter: Stephen O'Donnell Assignee: Stephen O'Donnell When a permission provider plugin is enabled (eg Ranger) there are some scenarios where it can throw a sub-class of an AccessControlException (eg RangerAccessControlException). If this exception is allowed to propagate up the stack, it can give problems in the HDFS Client, when it unwraps the remote exception containing the AccessControlException sub-class. Ideally, we should make AccessControlException final so it cannot be sub-classed, but that would be a breaking change at this point. Therefore I believe the safest thing to do, is to catch any AccessControlException that comes out of the permission enforcer plugin, and re-throw an AccessControlException instead. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16258) HDFS-13671 breaks TestBlockManager in branch-3.2
[ https://issues.apache.org/jira/browse/HDFS-16258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-16258: --- Description: TestBlockManager in branch-3.2 has two failed tests: * testDeleteCorruptReplicaWithStatleStorages * testBlockManagerMachinesArray Looks like broken by HDFS-13671. CC: [~brahmareddy] Branch-3.3 seems fine. was: TestBlockManager in branch-3.2 has two failed tests: * testDeleteCorruptReplicaWithStatleStorages * testBlockManagerMachinesArray Looks like broken by HDFS-13671. CC: [~brahmareddy] > HDFS-13671 breaks TestBlockManager in branch-3.2 > > > Key: HDFS-16258 > URL: https://issues.apache.org/jira/browse/HDFS-16258 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.2.3 >Reporter: Wei-Chiu Chuang >Priority: Blocker > > TestBlockManager in branch-3.2 has two failed tests: > * testDeleteCorruptReplicaWithStatleStorages > * testBlockManagerMachinesArray > Looks like broken by HDFS-13671. CC: [~brahmareddy] > Branch-3.3 seems fine. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16258) HDFS-13671 breaks TestBlockManager in branch-3.2
Wei-Chiu Chuang created HDFS-16258: -- Summary: HDFS-13671 breaks TestBlockManager in branch-3.2 Key: HDFS-16258 URL: https://issues.apache.org/jira/browse/HDFS-16258 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.2.3 Reporter: Wei-Chiu Chuang TestBlockManager in branch-3.2 has two failed tests: * testDeleteCorruptReplicaWithStatleStorages * testBlockManagerMachinesArray Looks like broken by HDFS-13671. CC: [~brahmareddy] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16257) [HDFS] [RBF] Guava cache performance issue in Router MountTableResolver
[ https://issues.apache.org/jira/browse/HDFS-16257?focusedWorklogId=660781=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660781 ] ASF GitHub Bot logged work on HDFS-16257: - Author: ASF GitHub Bot Created on: 06/Oct/21 08:57 Start Date: 06/Oct/21 08:57 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #3524: URL: https://github.com/apache/hadoop/pull/3524#issuecomment-935785881 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 9m 37s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ branch-2.10 Compile Tests _ | | +1 :green_heart: | mvninstall | 14m 34s | branch-2.10 passed | | +1 :green_heart: | compile | 0m 35s | branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | +1 :green_heart: | compile | 0m 29s | branch-2.10 passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 | | +1 :green_heart: | checkstyle | 0m 21s | branch-2.10 passed | | +1 :green_heart: | mvnsite | 0m 35s | branch-2.10 passed | | +1 :green_heart: | javadoc | 0m 49s | branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | +1 :green_heart: | javadoc | 0m 36s | branch-2.10 passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 | | +0 :ok: | spotbugs | 3m 32s | Both FindBugs and SpotBugs are enabled, using SpotBugs. | | +1 :green_heart: | spotbugs | 1m 2s | branch-2.10 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 28s | the patch passed | | +1 :green_heart: | compile | 0m 30s | the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | +1 :green_heart: | javac | 0m 30s | the patch passed | | +1 :green_heart: | compile | 0m 23s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 | | +1 :green_heart: | javac | 0m 23s | the patch passed | | +1 :green_heart: | checkstyle | 0m 15s | the patch passed | | +1 :green_heart: | mvnsite | 0m 28s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10 | | +1 :green_heart: | javadoc | 0m 30s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 | | +1 :green_heart: | spotbugs | 1m 7s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 17m 3s | hadoop-hdfs-rbf in the patch passed. | | +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. | | | | 54m 45s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3524/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/3524 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle | | uname | Linux ebc2d0291df7 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | branch-2.10 / dc03afc | | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 | | Multi-JDK versions | /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_262-b10 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3524/1/testReport/ | | Max. process+thread count | 1454 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3524/1/console | | versions | git=2.7.4 maven=3.3.9 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.
[jira] [Work logged] (HDFS-16251) Make hdfs_cat tool cross platform
[
https://issues.apache.org/jira/browse/HDFS-16251?focusedWorklogId=660778=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660778
]
ASF GitHub Bot logged work on HDFS-16251:
-
Author: ASF GitHub Bot
Created on: 06/Oct/21 08:47
Start Date: 06/Oct/21 08:47
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3523:
URL: https://github.com/apache/hadoop/pull/3523#issuecomment-935769933
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 11m 15s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 25m 19s | | trunk passed |
| +1 :green_heart: | compile | 2m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 53m 6s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 17s | | the patch passed |
| +1 :green_heart: | compile | 2m 27s | | the patch passed |
| +1 :green_heart: | cc | 2m 27s | | the patch passed |
| +1 :green_heart: | golang | 2m 27s | | the patch passed |
| +1 :green_heart: | javac | 2m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 47s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 31m 9s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 126m 23s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3523/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3523 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell golang |
| uname | Linux dc5738d38d86 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c57a63fbae71e00a84315643fdccf540f2baf026 |
| Default Java | Debian-11.0.12+7-post-Debian-2deb10u1 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3523/3/testReport/ |
| Max. process+thread count | 740 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3523/3/console |
| versions | git=2.20.1 maven=3.6.0 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 660778)
Time Spent: 50m (was: 40m)
> Make hdfs_cat tool cross platform
> -
>
> Key: HDFS-16251
> URL: https://issues.apache.org/jira/browse/HDFS-16251
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: libhdfs++, tools
>Affects Versions: 3.4.0
> Environment: Centos 7, Centos 8, Debian 10, Ubuntu Focal
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The source files for hdfs_cat uses *getopt* for parsing the command line
> arguments. getopt is available only on Linux and thus, isn't cross platform.
> We need to replace getopt with *boost::program_options* to make this cross
>
[jira] [Work logged] (HDFS-16257) [HDFS] [RBF] Guava cache performance issue in Router MountTableResolver
[ https://issues.apache.org/jira/browse/HDFS-16257?focusedWorklogId=660756=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660756 ] ASF GitHub Bot logged work on HDFS-16257: - Author: ASF GitHub Bot Created on: 06/Oct/21 08:02 Start Date: 06/Oct/21 08:02 Worklog Time Spent: 10m Work Description: symious commented on pull request #3524: URL: https://github.com/apache/hadoop/pull/3524#issuecomment-935700482 @goiri Could you help to review this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 660756) Time Spent: 20m (was: 10m) > [HDFS] [RBF] Guava cache performance issue in Router MountTableResolver > --- > > Key: HDFS-16257 > URL: https://issues.apache.org/jira/browse/HDFS-16257 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.10.1 >Reporter: Janus Chow >Assignee: Janus Chow >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Branch 2.10.1 uses guava version of 11.0.2, which has a bug which affects the > performance of cache, which was mentioned in HDFS-13821. > Since upgrading guava version seems affecting too much, this ticket is to add > a configuration setting when initializing cache to walk around this issue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16257) [HDFS] [RBF] Guava cache performance issue in Router MountTableResolver
[ https://issues.apache.org/jira/browse/HDFS-16257?focusedWorklogId=660754=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660754 ] ASF GitHub Bot logged work on HDFS-16257: - Author: ASF GitHub Bot Created on: 06/Oct/21 08:02 Start Date: 06/Oct/21 08:02 Worklog Time Spent: 10m Work Description: symious opened a new pull request #3524: URL: https://github.com/apache/hadoop/pull/3524 ### Description of PR Branch 2.10.1 uses guava version of 11.0.2, which has a bug which affects the performance of cache, which was mentioned in HDFS-13821. Since upgrading guava version seems affecting too much, this ticket is to add a configuration setting when initializing cache to walk around this issue. ### How was this patch tested? Locally tested. ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 660754) Remaining Estimate: 0h Time Spent: 10m > [HDFS] [RBF] Guava cache performance issue in Router MountTableResolver > --- > > Key: HDFS-16257 > URL: https://issues.apache.org/jira/browse/HDFS-16257 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.10.1 >Reporter: Janus Chow >Assignee: Janus Chow >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Branch 2.10.1 uses guava version of 11.0.2, which has a bug which affects the > performance of cache, which was mentioned in HDFS-13821. > Since upgrading guava version seems affecting too much, this ticket is to add > a configuration setting when initializing cache to walk around this issue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16257) [HDFS] [RBF] Guava cache performance issue in Router MountTableResolver
[ https://issues.apache.org/jira/browse/HDFS-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16257: -- Labels: pull-request-available (was: ) > [HDFS] [RBF] Guava cache performance issue in Router MountTableResolver > --- > > Key: HDFS-16257 > URL: https://issues.apache.org/jira/browse/HDFS-16257 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.10.1 >Reporter: Janus Chow >Assignee: Janus Chow >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Branch 2.10.1 uses guava version of 11.0.2, which has a bug which affects the > performance of cache, which was mentioned in HDFS-13821. > Since upgrading guava version seems affecting too much, this ticket is to add > a configuration setting when initializing cache to walk around this issue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13821) RBF: Add dfs.federation.router.mount-table.cache.enable so that users can disable cache
[
https://issues.apache.org/jira/browse/HDFS-13821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424499#comment-17424499
]
Janus Chow edited comment on HDFS-13821 at 10/6/21, 7:58 AM:
-
The performance bottleneck should be related to the bug mentioned in
[https://github.com/google/guava/issues/1055] .
We can work around this issue by setting initialCapacity to maxCacheSize
(mentioned in [https://unportant.info/chasing-down-guava-cache-slowness.html] )
In branch 2.10, the guava version is 11.0.2, it's still affected.
was (Author: symious):
The performance bottleneck should be related to the bug mentioned in
[https://github.com/google/guava/issues/1055.]
We can work around this issue by setting initialCapacity to maxCacheSize
(mentioned in [https://unportant.info/chasing-down-guava-cache-slowness.html).]
In branch 2.10, the guava version is 11.0.2, it's still affected.
> RBF: Add dfs.federation.router.mount-table.cache.enable so that users can
> disable cache
> ---
>
> Key: HDFS-13821
> URL: https://issues.apache.org/jira/browse/HDFS-13821
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.1.0, 2.9.1, 3.0.3
>Reporter: Hui Fei
>Assignee: Hui Fei
>Priority: Major
> Fix For: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2
>
> Attachments: HDFS-13821.001.patch, HDFS-13821.002.patch,
> HDFS-13821.003.patch, HDFS-13821.004.patch, HDFS-13821.005.patch,
> HDFS-13821.006.patch, HDFS-13821.007.patch, HDFS-13821.008.patch,
> LocalCacheTest.java, image-2018-08-13-11-27-49-023.png
>
>
> When i test rbf, if found performance problem.
> I found that ProxyAvgTime From Ganglia is so high, i run jstack on Router and
> get the following stack frames
> {quote}
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0005c264acd8> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
> at
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
> at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
> at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2249)
> at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
> at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
> at
> com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764)
> at
> org.apache.hadoop.hdfs.server.federation.resolver.MountTableResolver.getDestinationForPath(MountTableResolver.java:380)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getLocationsForPath(RouterRpcServer.java:2104)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getLocationsForPath(RouterRpcServer.java:2087)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getListing(RouterRpcServer.java:1050)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getListing(ClientNamenodeProtocolServerSideTranslatorPB.java:640)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> {quote}
> Many threads blocked on *LocalCache*
> After disable the cache, ProxyAvgTime is down as follow showed
> !image-2018-08-13-11-27-49-023.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16257) [HDFS] [RBF] Guava cache performance issue in Router MountTableResolver
Janus Chow created HDFS-16257: - Summary: [HDFS] [RBF] Guava cache performance issue in Router MountTableResolver Key: HDFS-16257 URL: https://issues.apache.org/jira/browse/HDFS-16257 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.10.1 Reporter: Janus Chow Assignee: Janus Chow Branch 2.10.1 uses guava version of 11.0.2, which has a bug which affects the performance of cache, which was mentioned in HDFS-13821. Since upgrading guava version seems affecting too much, this ticket is to add a configuration setting when initializing cache to walk around this issue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13821) RBF: Add dfs.federation.router.mount-table.cache.enable so that users can disable cache
[
https://issues.apache.org/jira/browse/HDFS-13821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424818#comment-17424818
]
Janus Chow commented on HDFS-13821:
---
Thanks [~elgoiri] for the comment.
{quote}Can you post a PR modifying these parameters?
{quote}
Sure, will raise another ticket for the parameter changes.
{quote}Overall, we are trying to get out of guava, is it possible here? I have
to say that the loading cache interface is pretty neat
{quote}
Personally, I do like using guava cache. But seems it's a little out of hand
for guava here. Since Router will cache each path, one namespace can have
millions of files. Router will forward to multi namespaces, so the hit rate
won't be too high, the operation of loadCache and expireCache would be some
overhead for this situation. I think the hit rate needs to be improved a lot.
> RBF: Add dfs.federation.router.mount-table.cache.enable so that users can
> disable cache
> ---
>
> Key: HDFS-13821
> URL: https://issues.apache.org/jira/browse/HDFS-13821
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.1.0, 2.9.1, 3.0.3
>Reporter: Hui Fei
>Assignee: Hui Fei
>Priority: Major
> Fix For: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2
>
> Attachments: HDFS-13821.001.patch, HDFS-13821.002.patch,
> HDFS-13821.003.patch, HDFS-13821.004.patch, HDFS-13821.005.patch,
> HDFS-13821.006.patch, HDFS-13821.007.patch, HDFS-13821.008.patch,
> LocalCacheTest.java, image-2018-08-13-11-27-49-023.png
>
>
> When i test rbf, if found performance problem.
> I found that ProxyAvgTime From Ganglia is so high, i run jstack on Router and
> get the following stack frames
> {quote}
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0005c264acd8> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
> at
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
> at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
> at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2249)
> at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
> at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
> at
> com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764)
> at
> org.apache.hadoop.hdfs.server.federation.resolver.MountTableResolver.getDestinationForPath(MountTableResolver.java:380)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getLocationsForPath(RouterRpcServer.java:2104)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getLocationsForPath(RouterRpcServer.java:2087)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getListing(RouterRpcServer.java:1050)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getListing(ClientNamenodeProtocolServerSideTranslatorPB.java:640)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> {quote}
> Many threads blocked on *LocalCache*
> After disable the cache, ProxyAvgTime is down as follow showed
> !image-2018-08-13-11-27-49-023.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16251) Make hdfs_cat tool cross platform
[
https://issues.apache.org/jira/browse/HDFS-16251?focusedWorklogId=660731=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660731
]
ASF GitHub Bot logged work on HDFS-16251:
-
Author: ASF GitHub Bot
Created on: 06/Oct/21 06:41
Start Date: 06/Oct/21 06:41
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3523:
URL: https://github.com/apache/hadoop/pull/3523#issuecomment-935583537
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 23m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 21m 14s | | trunk passed |
| +1 :green_heart: | compile | 2m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | | trunk passed |
| +1 :green_heart: | shadedclient | 43m 13s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 24s | | the patch passed |
| +1 :green_heart: | compile | 2m 37s | | the patch passed |
| +1 :green_heart: | cc | 2m 37s | | the patch passed |
| +1 :green_heart: | golang | 2m 37s | | the patch passed |
| +1 :green_heart: | javac | 2m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 9s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 32m 6s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not
generate ASF License warnings. |
| | | 123m 52s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3523/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3523 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell golang |
| uname | Linux 6a09341dfa92 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c57a63fbae71e00a84315643fdccf540f2baf026 |
| Default Java | Red Hat, Inc.-1.8.0_302-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3523/3/testReport/ |
| Max. process+thread count | 665 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3523/3/console |
| versions | git=2.27.0 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 660731)
Time Spent: 40m (was: 0.5h)
> Make hdfs_cat tool cross platform
> -
>
> Key: HDFS-16251
> URL: https://issues.apache.org/jira/browse/HDFS-16251
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: libhdfs++, tools
>Affects Versions: 3.4.0
> Environment: Centos 7, Centos 8, Debian 10, Ubuntu Focal
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The source files for hdfs_cat uses *getopt* for parsing the command line
> arguments. getopt is available only on Linux and thus, isn't cross platform.
> We need to replace getopt with *boost::program_options* to make this cross
>
