[jira] [Commented] (HDFS-15982) Deleted data on the Web UI must be saved to the trash
[ https://issues.apache.org/jira/browse/HDFS-15982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17324175#comment-17324175 ] Viraj Jasani commented on HDFS-15982: - FYI [~ayushtkn] [~weichiu] > Deleted data on the Web UI must be saved to the trash > -- > > Key: HDFS-15982 > URL: https://issues.apache.org/jira/browse/HDFS-15982 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs >Reporter: Bhavik Patel >Priority: Major > > If we delete the data from the Web UI then it should be first moved to > configured/default Trash directory and after the trash interval time, it > should be removed. currently, data directly removed from the system[This > behavior should be the same as CLI cmd] > > This can be helpful when the user accidentally deletes data from the Web UI. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15988) Stabilise HDFS Pre-Commit
[
https://issues.apache.org/jira/browse/HDFS-15988?focusedWorklogId=584582&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584582
]
ASF GitHub Bot logged work on HDFS-15988:
-
Author: ASF GitHub Bot
Created on: 17/Apr/21 02:37
Start Date: 17/Apr/21 02:37
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2860:
URL: https://github.com/apache/hadoop/pull/2860#issuecomment-821753013
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 50s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 26s | | trunk passed |
| +1 :green_heart: | compile | 4m 55s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 4m 32s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 21s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 15s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 30s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 18s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 6m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 36s | | the patch passed |
| +1 :green_heart: | compile | 4m 45s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 4m 45s | | the patch passed |
| +1 :green_heart: | compile | 4m 28s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 4m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 10s |
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2860/15/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
| hadoop-hdfs-project: The patch generated 1 new + 177 unchanged - 1 fixed =
178 total (was 178) |
| +1 :green_heart: | hadolint | 0m 2s | | No new issues. |
| +1 :green_heart: | mvnsite | 2m 43s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 2s | | No new issues. |
| +1 :green_heart: | javadoc | 2m 0s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 45s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 7m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 18s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 25s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 229m 5s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2860/15/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 18m 41s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 54s | | The patch does not
generate ASF License warnings. |
| | | 374m 14s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2860/15/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2860 |
| Optional Tests | dupname asflicense mvnsite un
[jira] [Work logged] (HDFS-15980) Fix tests for HDFS-15754 Create packet metrics for DataNode
[
https://issues.apache.org/jira/browse/HDFS-15980?focusedWorklogId=584579&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584579
]
ASF GitHub Bot logged work on HDFS-15980:
-
Author: ASF GitHub Bot
Created on: 17/Apr/21 01:21
Start Date: 17/Apr/21 01:21
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2921:
URL: https://github.com/apache/hadoop/pull/2921#issuecomment-821743561
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 52s | | trunk passed |
| +1 :green_heart: | compile | 1m 20s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 52s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 24s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 7s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 10s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 4s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 1m 4s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 52s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2921/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 6 new + 13 unchanged -
0 fixed = 19 total (was 13) |
| +1 :green_heart: | mvnsite | 1m 11s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 44s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 14s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 4s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 57s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 390m 24s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2921/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not
generate ASF License warnings. |
| | | 476m 50s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.TestDecommissioningStatus
|
| | hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks |
| | hadoop.hdfs.TestSnapshotCommands |
| | hadoop.hdfs.TestPersistBlocks |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList |
| | hadoop.hdfs.TestFileConcurrentReader |
| | hadoop.hdfs.TestStateAlignmentContextWithHA |
| | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
| |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.TestHDFSFileS
[jira] [Commented] (HDFS-15980) Fix tests for HDFS-15754 Create packet metrics for DataNode
[
https://issues.apache.org/jira/browse/HDFS-15980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17324142#comment-17324142
]
Hadoop QA commented on HDFS-15980:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
57s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} codespell {color} | {color:blue} 0m
0s{color} | | {color:blue} codespell was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | | {color:green} The patch appears to include 1 new or modified
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 33m
52s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 3m
6s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 7s{color} | | {color:green} branch has no errors when building and
testing our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
10s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green} 0m
0s{color} | | {color:green} The patch has no blanks issues. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 52s{color} |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt|https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2921/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt]
| {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 6 new +
13 unchanged - 0 fixed = 19 total (was 13) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 3m
4s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 57s{color} | | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Oth
[jira] [Commented] (HDFS-15878) Flaky test TestRouterWebHDFSContractCreate>AbstractContractCreateTest#testSyncable in Trunk
[ https://issues.apache.org/jira/browse/HDFS-15878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17324127#comment-17324127 ] Fengnan Li commented on HDFS-15878: --- [~inigoiri] How can we verify the tests are fixed? Is there some jenkins job I can monitor on? > Flaky test > TestRouterWebHDFSContractCreate>AbstractContractCreateTest#testSyncable in > Trunk > --- > > Key: HDFS-15878 > URL: https://issues.apache.org/jira/browse/HDFS-15878 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs, rbf >Reporter: Renukaprasad C >Assignee: Fengnan Li >Priority: Major > > ERROR] Tests run: 16, Failures: 0, Errors: 1, Skipped: 2, Time elapsed: > 24.627 s <<< FAILURE! - in > org.apache.hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate > [ERROR] > testSyncable(org.apache.hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate) > Time elapsed: 0.222 s <<< ERROR! > java.io.FileNotFoundException: File /test/testSyncable not found. > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) > at > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:110) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:576) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$900(WebHdfsFileSystem.java:146) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:892) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:858) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:652) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:690) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:686) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$ReadRunner.getRedirectedUrl(WebHdfsFileSystem.java:2307) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$ReadRunner.(WebHdfsFileSystem.java:2296) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$WebHdfsInputStream.(WebHdfsFileSystem.java:2176) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.open(WebHdfsFileSystem.java:1610) > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:975) > at > org.apache.hadoop.fs.contract.AbstractContractCreateTest.validateSyncableSemantics(AbstractContractCreateTest.java:556) > at > org.apache.hadoop.fs.contract.AbstractContractCreateTest.testSyncable(AbstractContractCreateTest.java:459) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.lang.Thread.run(Thread.java:748) > Caused by: > org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File > /test/testSyncable not found. > at > org.apache.
[jira] [Work logged] (HDFS-15970) Print network topology on the web
[
https://issues.apache.org/jira/browse/HDFS-15970?focusedWorklogId=584533&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584533
]
ASF GitHub Bot logged work on HDFS-15970:
-
Author: ASF GitHub Bot
Created on: 16/Apr/21 22:16
Start Date: 16/Apr/21 22:16
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2896:
URL: https://github.com/apache/hadoop/pull/2896#issuecomment-821608597
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 46s | | trunk passed |
| +1 :green_heart: | compile | 1m 30s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 56s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 51s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 22s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 1s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2896/5/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 54 new + 5 unchanged -
1 fixed = 59 total (was 6) |
| +1 :green_heart: | mvnsite | 1m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 21s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 34s | | the patch passed |
| +1 :green_heart: | shadedclient | 17m 43s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 239m 47s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2896/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 334m 20s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2896/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2896 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux c516aeb7f3b1 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28
05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4fc4bfa78aa50d0
[jira] [Work logged] (HDFS-15987) Improve oiv tool to parse fsimage file in parallel with delimited format
[
https://issues.apache.org/jira/browse/HDFS-15987?focusedWorklogId=584511&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584511
]
ASF GitHub Bot logged work on HDFS-15987:
-
Author: ASF GitHub Bot
Created on: 16/Apr/21 20:54
Start Date: 16/Apr/21 20:54
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2918:
URL: https://github.com/apache/hadoop/pull/2918#issuecomment-821559317
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 21m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 53s | | trunk passed |
| +1 :green_heart: | compile | 1m 31s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 26s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 9s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 4s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 28s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 9s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 16s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 16s | | the patch passed |
| +1 :green_heart: | compile | 1m 5s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 1m 5s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 56s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2918/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 13 new + 50 unchanged
- 0 fixed = 63 total (was 50) |
| +1 :green_heart: | mvnsite | 1m 15s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 46s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 16s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| -1 :x: | spotbugs | 3m 21s |
[/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2918/2/artifact/out/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html)
| hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 unchanged - 0 fixed = 2
total (was 0) |
| +1 :green_heart: | shadedclient | 19m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 323m 21s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2918/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 441m 33s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Found reliance on default encoding in
org.apache.hadoop.hdfs.tools.offlineImageViewer.PBImageTextWriter.outputInParallel(Configuration,
FsImageProto$FileSummary, ArrayList):in
org.apache.hadoop.hdfs.tools.offlineImageViewer.PBImageTextWriter.outputInParallel(Configuration,
FsImageProto$FileSummary, ArrayList): new java.io.PrintStream(String) At
PBImageTextWriter.java:[line 788] |
| | Exceptional return value of java.io.File.delete() ignored in
org.apache.hadoop.hdfs.tools.offlineImageViewer.PBImageTextWriter.mergeFiles(String[],
String) At PBImageTextWriter.java:ignored in
org.apache.hadoop.hdfs.tools.offlineImageV
[jira] [Work logged] (HDFS-15988) Stabilise HDFS Pre-Commit
[
https://issues.apache.org/jira/browse/HDFS-15988?focusedWorklogId=584510&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584510
]
ASF GitHub Bot logged work on HDFS-15988:
-
Author: ASF GitHub Bot
Created on: 16/Apr/21 20:52
Start Date: 16/Apr/21 20:52
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2860:
URL: https://github.com/apache/hadoop/pull/2860#issuecomment-821556485
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 34s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 52s | | trunk passed |
| +1 :green_heart: | compile | 4m 56s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 4m 33s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 19s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 10s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 29s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 16s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 6m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 26s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 39s | | the patch passed |
| +1 :green_heart: | compile | 4m 42s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 4m 42s | | the patch passed |
| +1 :green_heart: | compile | 4m 28s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 4m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 10s |
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2860/14/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
| hadoop-hdfs-project: The patch generated 1 new + 177 unchanged - 1 fixed =
178 total (was 178) |
| +1 :green_heart: | hadolint | 0m 2s | | No new issues. |
| +1 :green_heart: | mvnsite | 2m 46s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 2m 0s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 48s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 7m 0s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 7s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 25s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 238m 28s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2860/14/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 18m 54s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not
generate ASF License warnings. |
| | | 383m 44s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks |
| | hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
[jira] [Comment Edited] (HDFS-15982) Deleted data on the Web UI must be saved to the trash
[ https://issues.apache.org/jira/browse/HDFS-15982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17324072#comment-17324072 ] Viraj Jasani edited comment on HDFS-15982 at 4/16/21, 8:47 PM: --- +1 to this improvement. Given the importance, does it make sense to keep "moving file to .Trash" logic as part of FileSystem's Delete implementation rather than keeping at individual client interface (shell, webhdfs etc)? was (Author: vjasani): +1 to this improvement. Does it make sense to keep "moving file to .Trash" logic as part of FileSystem's Delete implementation rather than keeping at individual client interface (shell, webhdfs etc)? > Deleted data on the Web UI must be saved to the trash > -- > > Key: HDFS-15982 > URL: https://issues.apache.org/jira/browse/HDFS-15982 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs >Reporter: Bhavik Patel >Priority: Major > > If we delete the data from the Web UI then it should be first moved to > configured/default Trash directory and after the trash interval time, it > should be removed. currently, data directly removed from the system[This > behavior should be the same as CLI cmd] > > This can be helpful when the user accidentally deletes data from the Web UI. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15982) Deleted data on the Web UI must be saved to the trash
[ https://issues.apache.org/jira/browse/HDFS-15982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17324072#comment-17324072 ] Viraj Jasani commented on HDFS-15982: - +1 to this improvement. Does it make sense to keep "moving file to .Trash" logic as part of FileSystem's Delete implementation rather than keeping at individual client interface (shell, webhdfs etc)? > Deleted data on the Web UI must be saved to the trash > -- > > Key: HDFS-15982 > URL: https://issues.apache.org/jira/browse/HDFS-15982 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs >Reporter: Bhavik Patel >Priority: Major > > If we delete the data from the Web UI then it should be first moved to > configured/default Trash directory and after the trash interval time, it > should be removed. currently, data directly removed from the system[This > behavior should be the same as CLI cmd] > > This can be helpful when the user accidentally deletes data from the Web UI. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15988) Stabilise HDFS Pre-Commit
[ https://issues.apache.org/jira/browse/HDFS-15988?focusedWorklogId=584487&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584487 ] ASF GitHub Bot logged work on HDFS-15988: - Author: ASF GitHub Bot Created on: 16/Apr/21 20:24 Start Date: 16/Apr/21 20:24 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2860: URL: https://github.com/apache/hadoop/pull/2860#issuecomment-821541081 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2860/15/console in case of problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 584487) Time Spent: 20m (was: 10m) > Stabilise HDFS Pre-Commit > - > > Key: HDFS-15988 > URL: https://issues.apache.org/jira/browse/HDFS-15988 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Fix couple of unit-tests: > TestRouterRpc > TestRouterRpcMultiDest > TestNestedSnapshots > TestPersistBlocks > TestDirectoryScanner > * Increase Maven OPTS, Remove timeouts from couple of tests and Add a retry > flaky test option in the build, So, as to make the build little stable -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15979) Move within EZ fails and cannot remove nested EZs
[
https://issues.apache.org/jira/browse/HDFS-15979?focusedWorklogId=584405&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584405
]
ASF GitHub Bot logged work on HDFS-15979:
-
Author: ASF GitHub Bot
Created on: 16/Apr/21 17:55
Start Date: 16/Apr/21 17:55
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2919:
URL: https://github.com/apache/hadoop/pull/2919#issuecomment-821354712
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 19s | | trunk passed |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 19s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 32s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 26s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 26s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 26s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 58s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 18s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 28s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 13s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 243m 41s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2919/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not
generate ASF License warnings. |
| | | 338m 35s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.TestFileCreation |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2919/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2919 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 58fe044e51b6 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 687a324ecbe2a53006aeff1185ec106311555b8e |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-
[jira] [Updated] (HDFS-15980) Fix tests for HDFS-15754 Create packet metrics for DataNode
[ https://issues.apache.org/jira/browse/HDFS-15980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated HDFS-15980: -- Fix Version/s: 3.4.0 Affects Version/s: 3.4.0 Status: Patch Available (was: In Progress) https://github.com/apache/hadoop/pull/2921 > Fix tests for HDFS-15754 Create packet metrics for DataNode > --- > > Key: HDFS-15980 > URL: https://issues.apache.org/jira/browse/HDFS-15980 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.4.0 >Reporter: Zheng Shao >Assignee: Zheng Shao >Priority: Trivial > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 10m > Remaining Estimate: 0h > > HDFS-15754 introduces 4 new metrics in DataNodeMetrics. However the test > associated with the patch has some bugs. This issue is to fix those bugs in > the tests. > Please note that the non-test code of HDFS-15754 worked fine without any bugs. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15980) Fix tests for HDFS-15754 Create packet metrics for DataNode
[ https://issues.apache.org/jira/browse/HDFS-15980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-15980: -- Labels: pull-request-available (was: ) > Fix tests for HDFS-15754 Create packet metrics for DataNode > --- > > Key: HDFS-15980 > URL: https://issues.apache.org/jira/browse/HDFS-15980 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Zheng Shao >Assignee: Zheng Shao >Priority: Trivial > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > HDFS-15754 introduces 4 new metrics in DataNodeMetrics. However the test > associated with the patch has some bugs. This issue is to fix those bugs in > the tests. > Please note that the non-test code of HDFS-15754 worked fine without any bugs. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15980) Fix tests for HDFS-15754 Create packet metrics for DataNode
[ https://issues.apache.org/jira/browse/HDFS-15980?focusedWorklogId=584393&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584393 ] ASF GitHub Bot logged work on HDFS-15980: - Author: ASF GitHub Bot Created on: 16/Apr/21 17:23 Start Date: 16/Apr/21 17:23 Worklog Time Spent: 10m Work Description: uzshao opened a new pull request #2921: URL: https://github.com/apache/hadoop/pull/2921 See https://issues.apache.org/jira/browse/HDFS-15980 HDFS-15754 introduces 4 new metrics in DataNodeMetrics. However the test associated with the patch has some bugs. This issue is to fix those bugs in the tests. Please note that the non-test code of HDFS-15754 worked fine without any bugs. There are 3 issues: 1. The metric names in the tests were incorrect; 2. The tests only checked the metrics of DataNode 0, but one of the metrics, PacketsSlowWriteToMirror, is only updated on 2 of the 3 DataNodes, so this created an indeterministic failure. 3. The metric PacketsSlowWriteToOsCache was not updated in the test due to the fact that the size of the file was smaller than BlockReceiver.CACHE_DROP_LAG_BYTES All of them are fixed in this patch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 584393) Remaining Estimate: 0h Time Spent: 10m > Fix tests for HDFS-15754 Create packet metrics for DataNode > --- > > Key: HDFS-15980 > URL: https://issues.apache.org/jira/browse/HDFS-15980 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Zheng Shao >Assignee: Zheng Shao >Priority: Trivial > Time Spent: 10m > Remaining Estimate: 0h > > HDFS-15754 introduces 4 new metrics in DataNodeMetrics. However the test > associated with the patch has some bugs. This issue is to fix those bugs in > the tests. > Please note that the non-test code of HDFS-15754 worked fine without any bugs. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Reopened] (HDFS-15971) Make mkstemp cross platform
[ https://issues.apache.org/jira/browse/HDFS-15971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger reopened HDFS-15971: > Make mkstemp cross platform > --- > > Key: HDFS-15971 > URL: https://issues.apache.org/jira/browse/HDFS-15971 > Project: Hadoop HDFS > Issue Type: Improvement > Components: libhdfs++ >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h > Remaining Estimate: 0h > > mkstemp isn't available in Visual C++. Need to make it cross platform. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15971) Make mkstemp cross platform
[ https://issues.apache.org/jira/browse/HDFS-15971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated HDFS-15971: --- Fix Version/s: (was: 3.4.0) I've reverted this from trunk > Make mkstemp cross platform > --- > > Key: HDFS-15971 > URL: https://issues.apache.org/jira/browse/HDFS-15971 > Project: Hadoop HDFS > Issue Type: Improvement > Components: libhdfs++ >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > mkstemp isn't available in Visual C++. Need to make it cross platform. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-15980) Fix tests for HDFS-15754 Create packet metrics for DataNode
[ https://issues.apache.org/jira/browse/HDFS-15980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-15980 started by Zheng Shao. - > Fix tests for HDFS-15754 Create packet metrics for DataNode > --- > > Key: HDFS-15980 > URL: https://issues.apache.org/jira/browse/HDFS-15980 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Zheng Shao >Assignee: Zheng Shao >Priority: Trivial > > HDFS-15754 introduces 4 new metrics in DataNodeMetrics. However the test > associated with the patch has some bugs. This issue is to fix those bugs in > the tests. > Please note that the non-test code of HDFS-15754 worked fine without any bugs. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15971) Make mkstemp cross platform
[ https://issues.apache.org/jira/browse/HDFS-15971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17323953#comment-17323953 ] Eric Badger commented on HDFS-15971: Yea, I think reverting would be best until we can figure out how to fix it on RHEL. I'll revert it. I'm not familiar with the code that was modified, but I'm happy to test any patches on RHEL to make sure that they work on that environment before we merge again. > Make mkstemp cross platform > --- > > Key: HDFS-15971 > URL: https://issues.apache.org/jira/browse/HDFS-15971 > Project: Hadoop HDFS > Issue Type: Improvement > Components: libhdfs++ >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h > Remaining Estimate: 0h > > mkstemp isn't available in Visual C++. Need to make it cross platform. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15971) Make mkstemp cross platform
[ https://issues.apache.org/jira/browse/HDFS-15971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17323948#comment-17323948 ] Gautham Banasandra edited comment on HDFS-15971 at 4/16/21, 5:08 PM: - Sorry about the inconvenience [~ebadger]. It's quite strange as to why the CMake in RHEL 7.6 isn't able to find x_platform_obj_c_api. Since, this runs without any issues on Ubuntu Focal. [~elgoiri] I've started looking into this. But if it's blocking folks on RHEL 7, please feel free to revert this PR. was (Author: gautham): Sorry about the inconvenience [~ebadger]. It's quite strange as to why the CMake in RHEL 7.6 isn't able to find x_platform_obj_c_api. Since, this runs without any issues on Ubuntu Focal. [~elgoiri] I'll fix this over the weekend. But if it's blocking folks on RHEL 7, please feel free to revert this. > Make mkstemp cross platform > --- > > Key: HDFS-15971 > URL: https://issues.apache.org/jira/browse/HDFS-15971 > Project: Hadoop HDFS > Issue Type: Improvement > Components: libhdfs++ >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h > Remaining Estimate: 0h > > mkstemp isn't available in Visual C++. Need to make it cross platform. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15975) Use LongAdder instead of AtomicLong
[
https://issues.apache.org/jira/browse/HDFS-15975?focusedWorklogId=584372&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584372
]
ASF GitHub Bot logged work on HDFS-15975:
-
Author: ASF GitHub Bot
Created on: 16/Apr/21 17:05
Start Date: 16/Apr/21 17:05
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2907:
URL: https://github.com/apache/hadoop/pull/2907#issuecomment-821314734
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 36s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 16s | | trunk passed |
| +1 :green_heart: | compile | 20m 46s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 18m 23s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 3m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 14s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 2s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 4m 7s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 8m 15s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 26s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 51s | | the patch passed |
| +1 :green_heart: | compile | 20m 12s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 20m 12s | | the patch passed |
| +1 :green_heart: | compile | 18m 14s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 18m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 46s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2907/4/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 5 new + 242 unchanged - 5 fixed = 247 total (was
247) |
| +1 :green_heart: | mvnsite | 4m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 58s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 59s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 8m 56s | | the patch passed |
| +1 :green_heart: | shadedclient | 17m 39s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 33s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 31s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 242m 3s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2907/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not
generate ASF License warnings. |
| | | 461m 25s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.TestViewDistributedFileSystem |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2907/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2907 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsi
[jira] [Commented] (HDFS-15971) Make mkstemp cross platform
[ https://issues.apache.org/jira/browse/HDFS-15971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17323948#comment-17323948 ] Gautham Banasandra commented on HDFS-15971: --- Sorry about the inconvenience [~ebadger]. It's quite strange as to why the CMake in RHEL 7.6 isn't able to find x_platform_obj_c_api. Since, this runs without any issues on Ubuntu Focal. [~elgoiri] I'll fix this over the weekend. But if it's blocking folks on RHEL 7, please feel free to revert this. > Make mkstemp cross platform > --- > > Key: HDFS-15971 > URL: https://issues.apache.org/jira/browse/HDFS-15971 > Project: Hadoop HDFS > Issue Type: Improvement > Components: libhdfs++ >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h > Remaining Estimate: 0h > > mkstemp isn't available in Visual C++. Need to make it cross platform. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15985) Incorrect sorting will cause failure to load an FsImage file
[
https://issues.apache.org/jira/browse/HDFS-15985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17323944#comment-17323944
]
Xiaoqiao He commented on HDFS-15985:
I am also concerned that how to catch the exception here. which version
(with/without this patch) to checkpoint and generate fsimage and which
version(with/without this patch) to load it? While new version fsimage
format(after patch) could not been parsed by old version(without patch).
> Incorrect sorting will cause failure to load an FsImage file
>
>
> Key: HDFS-15985
> URL: https://issues.apache.org/jira/browse/HDFS-15985
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> After we have introduced HDFS-14617 or HDFS-14771, when loading an fsimage
> file, the following error will pop up:
> 2021-04-15 17:21:17,868 [293072]-INFO [main:FSImage@784]-Planning to load
> image:
> FSImageFile(file=//hadoop/hdfs/namenode/current/fsimage_0,
> cpktTxId=0)
> 2021-04-15 17:25:53,288 [568492]-INFO
> [main:FSImageFormatPBINode$Loader@229]-Loading 725097952 INodes.
> 2021-04-15 17:25:53,289 [568493]-ERROR [main:FSImage@730]-Failed to load
> image from
> FSImageFile(file=//hadoop/hdfs/namenode/current/fsimage_0,
> cpktTxId=0)
> java.lang.IllegalStateException: GLOBAL: serial number 3 does not exist
> at
> org.apache.hadoop.hdfs.server.namenode.SerialNumberMap.get(SerialNumberMap.java:85)
> at
> org.apache.hadoop.hdfs.server.namenode.SerialNumberManager.getString(SerialNumberManager.java:121)
> at
> org.apache.hadoop.hdfs.server.namenode.SerialNumberManager.getString(SerialNumberManager.java:125)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeWithAdditionalFields$PermissionStatusFormat.toPermissionStatus(INodeWithAdditionalFields.java:86)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadPermission(FSImageFormatPBINode.java:93)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeFile(FSImageFormatPBINode.java:303)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINode(FSImageFormatPBINode.java:280)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeSection(FSImageFormatPBINode.java:237)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:237)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:176)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:226)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:937)
> It was found that this anomaly was related to sorting, as follows:
> ArrayList sections = Lists.newArrayList(summary
> .getSectionsList());
> Collections.sort(sections, new Comparator() {
> @Override
> public int compare(FileSummary.Section s1, FileSummary.Section s2) {
> SectionName n1 = SectionName.fromString(s1.getName());
> SectionName n2 = SectionName.fromString(s2.getName());
> if (n1 == null) {
> return n2 == null? 0: -1;
> } else if (n2 == null) {
> return -1;
> } else {
> return n1.ordinal()-n2.ordinal();
> }
> }
> });
> When n1 != null and n2 == null, this will cause sorting errors.
> When loading Sections, the correct order of loading Sections:
> NS_INFO -> STRING_TABLE -> INODE
> If the sorting is incorrect, the loading order is as follows:
> INDOE -> NS_INFO -> STRING_TABLE
> Because when loading INODE, you need to rely on STRING_TABLE.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15970) Print network topology on the web
[
https://issues.apache.org/jira/browse/HDFS-15970?focusedWorklogId=584366&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584366
]
ASF GitHub Bot logged work on HDFS-15970:
-
Author: ASF GitHub Bot
Created on: 16/Apr/21 16:53
Start Date: 16/Apr/21 16:53
Worklog Time Spent: 10m
Work Description: tomscut commented on a change in pull request #2896:
URL: https://github.com/apache/hadoop/pull/2896#discussion_r614993153
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NetworkTopologyServlet.java
##
@@ -0,0 +1,115 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.namenode;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.hdfs.server.blockmanagement.BlockManager;
+import org.apache.hadoop.net.NetUtils;
+import org.apache.hadoop.net.Node;
+import org.apache.hadoop.net.NodeBase;
+import org.apache.hadoop.util.StringUtils;
+
+import javax.servlet.ServletContext;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import java.io.IOException;
+import java.io.PrintStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.TreeSet;
+
+/**
+ * A servlet to print out the network topology.
+ */
+@InterfaceAudience.Private
+public class NetworkTopologyServlet extends DfsServlet {
+
+ public static final String PATH_SPEC = "/topology";
+
+ @Override
+ public void doGet(HttpServletRequest request, HttpServletResponse response)
Review comment:
> Right now we are exposing this as plain text in the Web UI when
clicking.
> We could have something with some more format like a JSON that can be
shown nicely in the Web UI.
Thanks @goiri for your advice. I changed the code to support json format. By
default, the output is still in text format. When the request
header(Accept:application/json) is set, the results are displayed in JSON
format, just like conf api (http://namenode:port/conf).
Text format:
curl http://namenode:port/topology
Json format:
curl -H 'Accept:application/json' http://namenode:port/topology
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 584366)
Time Spent: 2.5h (was: 2h 20m)
> Print network topology on the web
> -
>
> Key: HDFS-15970
> URL: https://issues.apache.org/jira/browse/HDFS-15970
> Project: Hadoop HDFS
> Issue Type: Wish
>Reporter: tomscut
>Assignee: tomscut
>Priority: Minor
> Labels: pull-request-available
> Attachments: hdfs-topology-json.jpg, hdfs-topology.jpg, hdfs-web.jpg
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> In order to query the network topology information conveniently, we can print
> it on the web.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15970) Print network topology on the web
[ https://issues.apache.org/jira/browse/HDFS-15970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tomscut updated HDFS-15970: --- Attachment: hdfs-topology-json.jpg > Print network topology on the web > - > > Key: HDFS-15970 > URL: https://issues.apache.org/jira/browse/HDFS-15970 > Project: Hadoop HDFS > Issue Type: Wish >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Attachments: hdfs-topology-json.jpg, hdfs-topology.jpg, hdfs-web.jpg > > Time Spent: 2h 20m > Remaining Estimate: 0h > > In order to query the network topology information conveniently, we can print > it on the web. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15963) Unreleased volume references cause an infinite loop
[
https://issues.apache.org/jira/browse/HDFS-15963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He updated HDFS-15963:
---
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks [~zhangshuyan] for your report and contribution!
Thanks [~weichiu] for your reviews.
> Unreleased volume references cause an infinite loop
> ---
>
> Key: HDFS-15963
> URL: https://issues.apache.org/jira/browse/HDFS-15963
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Shuyan Zhang
>Assignee: Shuyan Zhang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: HDFS-15963.001.patch, HDFS-15963.002.patch,
> HDFS-15963.003.patch
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> When BlockSender throws an exception because the meta-data cannot be found,
> the volume reference obtained by the thread is not released, which causes the
> thread trying to remove the volume to wait and fall into an infinite loop.
> {code:java}
> boolean checkVolumesRemoved() {
> Iterator it = volumesBeingRemoved.iterator();
> while (it.hasNext()) {
> FsVolumeImpl volume = it.next();
> if (!volume.checkClosed()) {
> return false;
> }
> it.remove();
> }
> return true;
> }
> boolean checkClosed() {
> // always be true.
> if (this.reference.getReferenceCount() > 0) {
> FsDatasetImpl.LOG.debug("The reference count for {} is {}, wait to be 0.",
> this, reference.getReferenceCount());
> return false;
> }
> return true;
> }
> {code}
> At the same time, because the thread has been holding checkDirsLock when
> removing the volume, other threads trying to acquire the same lock will be
> permanently blocked.
> Similar problems also occur in RamDiskAsyncLazyPersistService and
> FsDatasetAsyncDiskService.
> This patch releases the three previously unreleased volume references.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15963) Unreleased volume references cause an infinite loop
[
https://issues.apache.org/jira/browse/HDFS-15963?focusedWorklogId=584335&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584335
]
ASF GitHub Bot logged work on HDFS-15963:
-
Author: ASF GitHub Bot
Created on: 16/Apr/21 16:11
Start Date: 16/Apr/21 16:11
Worklog Time Spent: 10m
Work Description: Hexiaoqiao commented on pull request #2889:
URL: https://github.com/apache/hadoop/pull/2889#issuecomment-821284155
Committed to trunk. Thanks @zhangshuyan0 for your works! Thanks @jojochuang
for your reviews!
Will backport to other active branches for a while.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 584335)
Time Spent: 4h (was: 3h 50m)
> Unreleased volume references cause an infinite loop
> ---
>
> Key: HDFS-15963
> URL: https://issues.apache.org/jira/browse/HDFS-15963
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Shuyan Zhang
>Assignee: Shuyan Zhang
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-15963.001.patch, HDFS-15963.002.patch,
> HDFS-15963.003.patch
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> When BlockSender throws an exception because the meta-data cannot be found,
> the volume reference obtained by the thread is not released, which causes the
> thread trying to remove the volume to wait and fall into an infinite loop.
> {code:java}
> boolean checkVolumesRemoved() {
> Iterator it = volumesBeingRemoved.iterator();
> while (it.hasNext()) {
> FsVolumeImpl volume = it.next();
> if (!volume.checkClosed()) {
> return false;
> }
> it.remove();
> }
> return true;
> }
> boolean checkClosed() {
> // always be true.
> if (this.reference.getReferenceCount() > 0) {
> FsDatasetImpl.LOG.debug("The reference count for {} is {}, wait to be 0.",
> this, reference.getReferenceCount());
> return false;
> }
> return true;
> }
> {code}
> At the same time, because the thread has been holding checkDirsLock when
> removing the volume, other threads trying to acquire the same lock will be
> permanently blocked.
> Similar problems also occur in RamDiskAsyncLazyPersistService and
> FsDatasetAsyncDiskService.
> This patch releases the three previously unreleased volume references.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15963) Unreleased volume references cause an infinite loop
[
https://issues.apache.org/jira/browse/HDFS-15963?focusedWorklogId=584332&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584332
]
ASF GitHub Bot logged work on HDFS-15963:
-
Author: ASF GitHub Bot
Created on: 16/Apr/21 16:08
Start Date: 16/Apr/21 16:08
Worklog Time Spent: 10m
Work Description: Hexiaoqiao merged pull request #2889:
URL: https://github.com/apache/hadoop/pull/2889
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 584332)
Time Spent: 3h 50m (was: 3h 40m)
> Unreleased volume references cause an infinite loop
> ---
>
> Key: HDFS-15963
> URL: https://issues.apache.org/jira/browse/HDFS-15963
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Shuyan Zhang
>Assignee: Shuyan Zhang
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-15963.001.patch, HDFS-15963.002.patch,
> HDFS-15963.003.patch
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> When BlockSender throws an exception because the meta-data cannot be found,
> the volume reference obtained by the thread is not released, which causes the
> thread trying to remove the volume to wait and fall into an infinite loop.
> {code:java}
> boolean checkVolumesRemoved() {
> Iterator it = volumesBeingRemoved.iterator();
> while (it.hasNext()) {
> FsVolumeImpl volume = it.next();
> if (!volume.checkClosed()) {
> return false;
> }
> it.remove();
> }
> return true;
> }
> boolean checkClosed() {
> // always be true.
> if (this.reference.getReferenceCount() > 0) {
> FsDatasetImpl.LOG.debug("The reference count for {} is {}, wait to be 0.",
> this, reference.getReferenceCount());
> return false;
> }
> return true;
> }
> {code}
> At the same time, because the thread has been holding checkDirsLock when
> removing the volume, other threads trying to acquire the same lock will be
> permanently blocked.
> Similar problems also occur in RamDiskAsyncLazyPersistService and
> FsDatasetAsyncDiskService.
> This patch releases the three previously unreleased volume references.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15974) RBF: Unable to display the datanode UI of the router
[
https://issues.apache.org/jira/browse/HDFS-15974?focusedWorklogId=584316&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584316
]
ASF GitHub Bot logged work on HDFS-15974:
-
Author: ASF GitHub Bot
Created on: 16/Apr/21 15:40
Start Date: 16/Apr/21 15:40
Worklog Time Spent: 10m
Work Description: goiri commented on a change in pull request #2915:
URL: https://github.com/apache/hadoop/pull/2915#discussion_r614937252
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/metrics/NamenodeBeanMetrics.java
##
@@ -855,7 +855,7 @@ public long getNumberOfSnapshottableDirs() {
@Override
public String getEnteringMaintenanceNodes() {
-return "N/A";
+return null;
Review comment:
Can we add a test?
I don't think we can reproduce the JSON parsing in the browser side but
let's do something closer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 584316)
Time Spent: 20m (was: 10m)
> RBF: Unable to display the datanode UI of the router
>
>
> Key: HDFS-15974
> URL: https://issues.apache.org/jira/browse/HDFS-15974
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf, ui
>Affects Versions: 3.4.0
>Reporter: zhu
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-15358-1.patch, image-2021-04-15-11-36-47-644.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Clicking the Datanodes tag on the Router UI does not respond.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15987) Improve oiv tool to parse fsimage file in parallel with delimited format
[
https://issues.apache.org/jira/browse/HDFS-15987?focusedWorklogId=584300&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584300
]
ASF GitHub Bot logged work on HDFS-15987:
-
Author: ASF GitHub Bot
Created on: 16/Apr/21 14:53
Start Date: 16/Apr/21 14:53
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2918:
URL: https://github.com/apache/hadoop/pull/2918#issuecomment-821233568
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 33s | | trunk passed |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 18s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 2s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 29s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 21s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2918/1/artifact/out/blanks-eol.txt)
| The patch has 5 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 0m 57s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2918/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 13 new + 51 unchanged
- 0 fixed = 64 total (was 51) |
| +1 :green_heart: | mvnsite | 1m 17s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 52s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 25s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| -1 :x: | spotbugs | 3m 35s |
[/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2918/1/artifact/out/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html)
| hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 unchanged - 0 fixed = 2
total (was 0) |
| +1 :green_heart: | shadedclient | 18m 15s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 237m 43s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2918/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 333m 11s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Found reliance on default encoding in
org.apache.hadoop.hdfs.tools.offlineImageViewer.PBImageTextWriter.outputInParallel(Configuration,
FsImageProto$FileSummary, ArrayList):in
org.apache.hadoop.hdfs.tools.offlineImageViewer.PBImageTextWriter.outputInParallel(Configuration,
FsImageProto$FileSummary, ArrayList): new java.io.PrintStream(String) At
PBImageTextWriter.java:[line 788] |
| | Exceptional return value of java.io.File.delete
[jira] [Updated] (HDFS-15988) Stabilise HDFS Pre-Commit
[ https://issues.apache.org/jira/browse/HDFS-15988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-15988: -- Labels: pull-request-available (was: ) > Stabilise HDFS Pre-Commit > - > > Key: HDFS-15988 > URL: https://issues.apache.org/jira/browse/HDFS-15988 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Fix couple of unit-tests: > TestRouterRpc > TestRouterRpcMultiDest > TestNestedSnapshots > TestPersistBlocks > TestDirectoryScanner > * Increase Maven OPTS, Remove timeouts from couple of tests and Add a retry > flaky test option in the build, So, as to make the build little stable -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15988) Stabilise HDFS Pre-Commit
[ https://issues.apache.org/jira/browse/HDFS-15988?focusedWorklogId=584289&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584289 ] ASF GitHub Bot logged work on HDFS-15988: - Author: ASF GitHub Bot Created on: 16/Apr/21 14:30 Start Date: 16/Apr/21 14:30 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2860: URL: https://github.com/apache/hadoop/pull/2860#issuecomment-821218156 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2860/14/console in case of problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 584289) Remaining Estimate: 0h Time Spent: 10m > Stabilise HDFS Pre-Commit > - > > Key: HDFS-15988 > URL: https://issues.apache.org/jira/browse/HDFS-15988 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Fix couple of unit-tests: > TestRouterRpc > TestRouterRpcMultiDest > TestNestedSnapshots > TestPersistBlocks > TestDirectoryScanner > * Increase Maven OPTS, Remove timeouts from couple of tests and Add a retry > flaky test option in the build, So, as to make the build little stable -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15988) Stabilise HDFS Pre-Commit
[ https://issues.apache.org/jira/browse/HDFS-15988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15988: Status: Patch Available (was: Open) > Stabilise HDFS Pre-Commit > - > > Key: HDFS-15988 > URL: https://issues.apache.org/jira/browse/HDFS-15988 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > > Fix couple of unit-tests: > TestRouterRpc > TestRouterRpcMultiDest > TestNestedSnapshots > TestPersistBlocks > TestDirectoryScanner > * Increase Maven OPTS, Remove timeouts from couple of tests and Add a retry > flaky test option in the build, So, as to make the build little stable -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15988) Stabilise HDFS Pre-Commit
Ayush Saxena created HDFS-15988: --- Summary: Stabilise HDFS Pre-Commit Key: HDFS-15988 URL: https://issues.apache.org/jira/browse/HDFS-15988 Project: Hadoop HDFS Issue Type: Bug Reporter: Ayush Saxena Assignee: Ayush Saxena Fix couple of unit-tests: TestRouterRpc TestRouterRpcMultiDest TestNestedSnapshots TestPersistBlocks TestDirectoryScanner * Increase Maven OPTS, Remove timeouts from couple of tests and Add a retry flaky test option in the build, So, as to make the build little stable -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15957) The ignored IOException in the RPC response sent by FSEditLogAsync can cause the HDFS client to hang
[
https://issues.apache.org/jira/browse/HDFS-15957?focusedWorklogId=584274&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584274
]
ASF GitHub Bot logged work on HDFS-15957:
-
Author: ASF GitHub Bot
Created on: 16/Apr/21 14:05
Start Date: 16/Apr/21 14:05
Worklog Time Spent: 10m
Work Description: daryn-sharp commented on a change in pull request #2878:
URL: https://github.com/apache/hadoop/pull/2878#discussion_r614865039
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogAsync.java
##
@@ -378,13 +381,18 @@ public void logSyncWait() {
@Override
public void logSyncNotify(RuntimeException syncEx) {
- try {
-if (syncEx == null) {
- call.sendResponse();
-} else {
- call.abortResponse(syncEx);
+ for (int retries = 0; retries <= RESPONSE_SEND_RETRIES; retries++) {
Review comment:
> For example, for transient connection issues, retry would help ...
it's not like retry won't help with connection issue, it's just a matter of
whether our fix implements the retry correctly or not, i.e., whether we should
re-create a connection object or not.
There is no scenario in which an IOE is recoverable except in an artificial
fault injection condition. There is no "transient" failure writing to a
socket – it's game over. An IOE bubbling up to this point means the
underlying low-level ipc code has already closed the connection and simply
re-thrown the error to inform the caller.
Re-creating a connection object is fundamentally impossible. Ignoring that
impossibility, even if a _speculated_ future bug left the connection open it's
in an unknown/inconsistent state with possible partial data written so writing
anything more is a corrupted or duplicate response for the client.
> Therefore, I think it's still worth adding the retry logic here, although
it might not be able to handle the two scenarios you describe here. Do you
agree?
I 100% disagree with a broken solution in search of a non-existent problem.
_Any retry for any reason is inherently a bug_.
Which brings us back to:
> we are doing fault injection testing, and we inject an IOException in
call.sendResponse(), and then we observe the symptom that the client gets
stuck. In this scenario, retry can help.
This is a perfect illustration of the danger of using fault injection to
create an artificial bug, and then "solving" that artificial bug _with a real
bug_.
-1 on this patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 584274)
Time Spent: 1h 10m (was: 1h)
> The ignored IOException in the RPC response sent by FSEditLogAsync can cause
> the HDFS client to hang
>
>
> Key: HDFS-15957
> URL: https://issues.apache.org/jira/browse/HDFS-15957
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs async, namenode
>Affects Versions: 3.2.2
>Reporter: Haoze Wu
>Priority: Critical
> Labels: pull-request-available
> Attachments: fsshell.txt, namenode.txt, reproduce.patch,
> secondnamenode.txt
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> In FSEditLogAsync, the RpcEdit notification in line 248 could be skipped,
> because the possible exception (e.g., IOException) thrown in line 365 is
> always ignored.
>
> {code:java}
> //hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogAsync.java
> class FSEditLogAsync extends FSEditLog implements Runnable {
> // ...
> @Override
> public void run() {
> try {
> while (true) {
> boolean doSync;
> Edit edit = dequeueEdit();
> if (edit != null) {
> // sync if requested by edit log.
> doSync = edit.logEdit();
> syncWaitQ.add(edit);
> } else {
> // sync when editq runs dry, but have edits pending a sync.
> doSync = !syncWaitQ.isEmpty();
> }
> if (doSync) {
> // normally edit log exceptions cause the NN to terminate, but tests
> // relying on ExitUtil.terminate need to see the exception.
> RuntimeException syncEx = null;
> try {
> logSync(getLastWrittenTxId());
> } catch (RuntimeException ex) {
> syncEx = ex;
> }
> while ((edit = syncWaitQ.poll()) !
[jira] [Work logged] (HDFS-15985) Incorrect sorting will cause failure to load an FsImage file
[
https://issues.apache.org/jira/browse/HDFS-15985?focusedWorklogId=584254&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584254
]
ASF GitHub Bot logged work on HDFS-15985:
-
Author: ASF GitHub Bot
Created on: 16/Apr/21 13:15
Start Date: 16/Apr/21 13:15
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2917:
URL: https://github.com/apache/hadoop/pull/2917#issuecomment-821165566
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 16s | | trunk passed |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 56s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 26s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 18s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 10s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 6s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 1m 6s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 10s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 44s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 4s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 53s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 239m 19s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2917/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not
generate ASF License warnings. |
| | | 324m 21s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2917/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2917 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux f355de3bff5a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c82be48e9ae3807b12c62fd4c6797a8af6cca373 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openj
[jira] [Commented] (HDFS-15985) Incorrect sorting will cause failure to load an FsImage file
[
https://issues.apache.org/jira/browse/HDFS-15985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17323796#comment-17323796
]
Stephen O'Donnell commented on HDFS-15985:
--
[~jianghuazhu] Any idea on how a section name can be null at this stage? I
thought it was taking the list of sections from the image and hence they must
all have a name. Unless the image was created on a newer version with new
section names the namenode loading the image does not understand?
> Incorrect sorting will cause failure to load an FsImage file
>
>
> Key: HDFS-15985
> URL: https://issues.apache.org/jira/browse/HDFS-15985
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> After we have introduced HDFS-14617 or HDFS-14771, when loading an fsimage
> file, the following error will pop up:
> 2021-04-15 17:21:17,868 [293072]-INFO [main:FSImage@784]-Planning to load
> image:
> FSImageFile(file=//hadoop/hdfs/namenode/current/fsimage_0,
> cpktTxId=0)
> 2021-04-15 17:25:53,288 [568492]-INFO
> [main:FSImageFormatPBINode$Loader@229]-Loading 725097952 INodes.
> 2021-04-15 17:25:53,289 [568493]-ERROR [main:FSImage@730]-Failed to load
> image from
> FSImageFile(file=//hadoop/hdfs/namenode/current/fsimage_0,
> cpktTxId=0)
> java.lang.IllegalStateException: GLOBAL: serial number 3 does not exist
> at
> org.apache.hadoop.hdfs.server.namenode.SerialNumberMap.get(SerialNumberMap.java:85)
> at
> org.apache.hadoop.hdfs.server.namenode.SerialNumberManager.getString(SerialNumberManager.java:121)
> at
> org.apache.hadoop.hdfs.server.namenode.SerialNumberManager.getString(SerialNumberManager.java:125)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeWithAdditionalFields$PermissionStatusFormat.toPermissionStatus(INodeWithAdditionalFields.java:86)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadPermission(FSImageFormatPBINode.java:93)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeFile(FSImageFormatPBINode.java:303)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINode(FSImageFormatPBINode.java:280)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeSection(FSImageFormatPBINode.java:237)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:237)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:176)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:226)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:937)
> It was found that this anomaly was related to sorting, as follows:
> ArrayList sections = Lists.newArrayList(summary
> .getSectionsList());
> Collections.sort(sections, new Comparator() {
> @Override
> public int compare(FileSummary.Section s1, FileSummary.Section s2) {
> SectionName n1 = SectionName.fromString(s1.getName());
> SectionName n2 = SectionName.fromString(s2.getName());
> if (n1 == null) {
> return n2 == null? 0: -1;
> } else if (n2 == null) {
> return -1;
> } else {
> return n1.ordinal()-n2.ordinal();
> }
> }
> });
> When n1 != null and n2 == null, this will cause sorting errors.
> When loading Sections, the correct order of loading Sections:
> NS_INFO -> STRING_TABLE -> INODE
> If the sorting is incorrect, the loading order is as follows:
> INDOE -> NS_INFO -> STRING_TABLE
> Because when loading INODE, you need to rely on STRING_TABLE.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15981) Removing redundant block queues will slow down block reporting
[
https://issues.apache.org/jira/browse/HDFS-15981?focusedWorklogId=584252&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584252
]
ASF GitHub Bot logged work on HDFS-15981:
-
Author: ASF GitHub Bot
Created on: 16/Apr/21 13:08
Start Date: 16/Apr/21 13:08
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2916:
URL: https://github.com/apache/hadoop/pull/2916#issuecomment-821161737
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 1s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 54s | | trunk passed |
| +1 :green_heart: | compile | 1m 20s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 13s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 27s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 15s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 28s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 16s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 16s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 55s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 46s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 18s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 59s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 397m 32s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2916/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 17s | | The patch does not
generate ASF License warnings. |
| | | 492m 24s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.TestPersistBlocks |
| | hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.server.namenode.ha.TestStandbyIsHot |
| | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| | hadoop.hdfs.TestSafeMode |
| | hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.TestViewDistributedFileSystemWithMountLinks |
| | hadoop.hdfs.TestSnapshotCommands |
| |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.server.namenode.TestNamenodeStorageDirectives |
| | hadoop.hdfs.server.datanod
[jira] [Commented] (HDFS-15967) Improve the log for Short Circuit Local Reads
[
https://issues.apache.org/jira/browse/HDFS-15967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17323780#comment-17323780
]
Hadoop QA commented on HDFS-15967:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
25s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to
include any new or modified tests. Please justify why no new tests are needed
for this patch. Also please list what manual steps were performed to verify
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
24s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 54s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 23m
21s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 3m
12s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
16s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 12s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {col
[jira] [Resolved] (HDFS-15981) Removing redundant block queues will slow down block reporting
[ https://issues.apache.org/jira/browse/HDFS-15981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhanghuazong resolved HDFS-15981. - Resolution: Fixed > Removing redundant block queues will slow down block reporting > -- > > Key: HDFS-15981 > URL: https://issues.apache.org/jira/browse/HDFS-15981 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: zhanghuazong >Assignee: zhanghuazong >Priority: Major > Labels: pull-request-available > Attachments: HDFS-15981.patch > > Time Spent: 10m > Remaining Estimate: 0h > > When the block report satisfies the block distribution strategy, the block is > removed from the lowredundancyBlocks. But removing the block from the > lowredundancyBlocks is a redundant operation. > First, in the patch queue, the block removal operation will be performed in > the method chooseSourceDatanodes and validateReconstructionWork. > second, the removal of the block report will only be at the > QUEUE_REPLICAS_BADLY_DISTRIBUTED level, which is not an accurate operation. > Finally, when there is a large amount of data in the > QUEUE_REPLICAS_BADLY_DISTRIBUTED queue, the processing efficiency of the > block report will be reduced -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15979) Move within EZ fails and cannot remove nested EZs
[ https://issues.apache.org/jira/browse/HDFS-15979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-15979: -- Labels: pull-request-available (was: ) > Move within EZ fails and cannot remove nested EZs > - > > Key: HDFS-15979 > URL: https://issues.apache.org/jira/browse/HDFS-15979 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption, hdfs >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: pull-request-available > Attachments: HDFS-15979.001.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Moving between EZ directories should work fine if the EZ key for the > directories is identical. If the key is name identical then no > decrypt/re-encrypt is necessary. > However, the rename operation checks more than the key name. It compares the > inode number (unique identifier) of the source and dest dirs which will never > be the same for 2 dirs resulting in the cited failure. Note it also > incorrectly compares the key version. > A related issue is if an ancestor of a EZ share the same key (ie. > /projects/foo and /projects/foo/bar/blah both use same key), files also > cannot be moved from the child to a parent dir, plus the child EZ cannot be > removed even though it's now covered by the ancestor. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15979) Move within EZ fails and cannot remove nested EZs
[ https://issues.apache.org/jira/browse/HDFS-15979?focusedWorklogId=584239&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584239 ] ASF GitHub Bot logged work on HDFS-15979: - Author: ASF GitHub Bot Created on: 16/Apr/21 12:15 Start Date: 16/Apr/21 12:15 Worklog Time Spent: 10m Work Description: amahussein opened a new pull request #2919: URL: https://github.com/apache/hadoop/pull/2919 [HDFS-15979](https://issues.apache.org/jira/browse/HDFS-15979) Move within EZ fails and cannot remove nested EZs The test failures are not related to the changes. Those three tests have related opened Jiras and they were failing for some time. The changes were contributed by @daryn-sharp and we have our internal clusters running on those changes with hadoop-2.8 and hadoop-2.10. I made some modifications in order to handle the conflict since our internal branch has a feature that is not merged yet into the community (HDFS-13009). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 584239) Remaining Estimate: 0h Time Spent: 10m > Move within EZ fails and cannot remove nested EZs > - > > Key: HDFS-15979 > URL: https://issues.apache.org/jira/browse/HDFS-15979 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption, hdfs >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Attachments: HDFS-15979.001.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Moving between EZ directories should work fine if the EZ key for the > directories is identical. If the key is name identical then no > decrypt/re-encrypt is necessary. > However, the rename operation checks more than the key name. It compares the > inode number (unique identifier) of the source and dest dirs which will never > be the same for 2 dirs resulting in the cited failure. Note it also > incorrectly compares the key version. > A related issue is if an ancestor of a EZ share the same key (ie. > /projects/foo and /projects/foo/bar/blah both use same key), files also > cannot be moved from the child to a parent dir, plus the child EZ cannot be > removed even though it's now covered by the ancestor. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15967) Improve the log for Short Circuit Local Reads
[ https://issues.apache.org/jira/browse/HDFS-15967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17323764#comment-17323764 ] Bhavik Patel commented on HDFS-15967: - Thank you [~aajisaka] > Improve the log for Short Circuit Local Reads > - > > Key: HDFS-15967 > URL: https://issues.apache.org/jira/browse/HDFS-15967 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Bhavik Patel >Assignee: Bhavik Patel >Priority: Minor > Attachments: HDFS-15967.001.patch > > > Improve the log for Short Circuit Local Reads -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15968) Improve the log for The DecayRpcScheduler
[ https://issues.apache.org/jira/browse/HDFS-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhavik Patel updated HDFS-15968: Status: Patch Available (was: In Progress) > Improve the log for The DecayRpcScheduler > -- > > Key: HDFS-15968 > URL: https://issues.apache.org/jira/browse/HDFS-15968 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Bhavik Patel >Assignee: Bhavik Patel >Priority: Minor > Attachments: HDFS-15968.001.patch > > > Improve the log for The DecayRpcScheduler to make use of the SELF4j logger > factory -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-15968) Improve the log for The DecayRpcScheduler
[ https://issues.apache.org/jira/browse/HDFS-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-15968 started by Bhavik Patel. --- > Improve the log for The DecayRpcScheduler > -- > > Key: HDFS-15968 > URL: https://issues.apache.org/jira/browse/HDFS-15968 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Bhavik Patel >Assignee: Bhavik Patel >Priority: Minor > Attachments: HDFS-15968.001.patch > > > Improve the log for The DecayRpcScheduler to make use of the SELF4j logger > factory -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15968) Improve the log for The DecayRpcScheduler
[ https://issues.apache.org/jira/browse/HDFS-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhavik Patel updated HDFS-15968: Status: Open (was: Patch Available) > Improve the log for The DecayRpcScheduler > -- > > Key: HDFS-15968 > URL: https://issues.apache.org/jira/browse/HDFS-15968 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Bhavik Patel >Assignee: Bhavik Patel >Priority: Minor > Attachments: HDFS-15968.001.patch > > > Improve the log for The DecayRpcScheduler to make use of the SELF4j logger > factory -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15963) Unreleased volume references cause an infinite loop
[
https://issues.apache.org/jira/browse/HDFS-15963?focusedWorklogId=584234&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584234
]
ASF GitHub Bot logged work on HDFS-15963:
-
Author: ASF GitHub Bot
Created on: 16/Apr/21 12:03
Start Date: 16/Apr/21 12:03
Worklog Time Spent: 10m
Work Description: jojochuang commented on pull request #2889:
URL: https://github.com/apache/hadoop/pull/2889#issuecomment-82112
Go ahead. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 584234)
Time Spent: 3h 40m (was: 3.5h)
> Unreleased volume references cause an infinite loop
> ---
>
> Key: HDFS-15963
> URL: https://issues.apache.org/jira/browse/HDFS-15963
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Shuyan Zhang
>Assignee: Shuyan Zhang
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-15963.001.patch, HDFS-15963.002.patch,
> HDFS-15963.003.patch
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> When BlockSender throws an exception because the meta-data cannot be found,
> the volume reference obtained by the thread is not released, which causes the
> thread trying to remove the volume to wait and fall into an infinite loop.
> {code:java}
> boolean checkVolumesRemoved() {
> Iterator it = volumesBeingRemoved.iterator();
> while (it.hasNext()) {
> FsVolumeImpl volume = it.next();
> if (!volume.checkClosed()) {
> return false;
> }
> it.remove();
> }
> return true;
> }
> boolean checkClosed() {
> // always be true.
> if (this.reference.getReferenceCount() > 0) {
> FsDatasetImpl.LOG.debug("The reference count for {} is {}, wait to be 0.",
> this, reference.getReferenceCount());
> return false;
> }
> return true;
> }
> {code}
> At the same time, because the thread has been holding checkDirsLock when
> removing the volume, other threads trying to acquire the same lock will be
> permanently blocked.
> Similar problems also occur in RamDiskAsyncLazyPersistService and
> FsDatasetAsyncDiskService.
> This patch releases the three previously unreleased volume references.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15979) Move within EZ fails and cannot remove nested EZs
[ https://issues.apache.org/jira/browse/HDFS-15979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17323602#comment-17323602 ] Ahmed Hussein commented on HDFS-15979: -- The test failures are not related to the changes. Those three tests have related opened Jiras and they were failing for some time. The changes were contributed by [~daryn] and we have our internal clusters running on those changes with hadoop-2.8 and hadoop-2.10. I made some modifications in order to handle the conflict since our internal branch has a feature that is not merged yet into the community (HDFS-13009). I always struggle with the decision of providing patches Vs PR because reviewers have different preferences. Since [~weichiu] Is watching this jira (he prefers PRs as far as I remember), I am going to post a PR addressing the checkstyle warnings. > Move within EZ fails and cannot remove nested EZs > - > > Key: HDFS-15979 > URL: https://issues.apache.org/jira/browse/HDFS-15979 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption, hdfs >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Attachments: HDFS-15979.001.patch > > > Moving between EZ directories should work fine if the EZ key for the > directories is identical. If the key is name identical then no > decrypt/re-encrypt is necessary. > However, the rename operation checks more than the key name. It compares the > inode number (unique identifier) of the source and dest dirs which will never > be the same for 2 dirs resulting in the cited failure. Note it also > incorrectly compares the key version. > A related issue is if an ancestor of a EZ share the same key (ie. > /projects/foo and /projects/foo/bar/blah both use same key), files also > cannot be moved from the child to a parent dir, plus the child EZ cannot be > removed even though it's now covered by the ancestor. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15970) Print network topology on the web
[ https://issues.apache.org/jira/browse/HDFS-15970?focusedWorklogId=584171&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584171 ] ASF GitHub Bot logged work on HDFS-15970: - Author: ASF GitHub Bot Created on: 16/Apr/21 09:43 Start Date: 16/Apr/21 09:43 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #2896: URL: https://github.com/apache/hadoop/pull/2896#issuecomment-821056074 > Mostly looks good to me. Is it possible to extend to Router UI as @ayushtkn commented? (maybe another jira?) @tasanuma @ayushtkn I'll create another JIRA later to support the Router UI, what do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 584171) Time Spent: 2h 20m (was: 2h 10m) > Print network topology on the web > - > > Key: HDFS-15970 > URL: https://issues.apache.org/jira/browse/HDFS-15970 > Project: Hadoop HDFS > Issue Type: Wish >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Attachments: hdfs-topology.jpg, hdfs-web.jpg > > Time Spent: 2h 20m > Remaining Estimate: 0h > > In order to query the network topology information conveniently, we can print > it on the web. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15987) Improve oiv tool to parse fsimage file in parallel with delimited format
[
https://issues.apache.org/jira/browse/HDFS-15987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hui Fei reassigned HDFS-15987:
--
Assignee: Hongbing Wang
> Improve oiv tool to parse fsimage file in parallel with delimited format
>
>
> Key: HDFS-15987
> URL: https://issues.apache.org/jira/browse/HDFS-15987
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Hongbing Wang
>Assignee: Hongbing Wang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The purpose of this Jira is to improve oiv tool to parse fsimage file with
> sub-sections (see -HDFS-14617-) in parallel with delmited format.
> 1.Serial parsing is time-consuming
> The time to serially parse a large fsimage with delimited format (e.g. `hdfs
> oiv -p Delimited -t ...`) is as follows:
> {code:java}
> 1) Loading string table: -> Not time consuming.
> 2) Loading inode references: -> Not time consuming
> 3) Loading directories in INode section: -> Slightly time consuming (3%)
> 4) Loading INode directory section: -> A bit time consuming (11%)
> 5) Output: -> Very time consuming (86%){code}
> Therefore, output is the most parallelized stage.
> 2.How to output in parallel
> The sub-sections are grouped in order, and each thread processes a group and
> outputs it to the file corresponding to each thread, and finally merges the
> output files.
> 3. The result of a test
> {code:java}
> input fsimage file info:
> 3.4G, 12 sub-sections, 55976500 INodes
> -
> Threads TotalTime OutputTime MergeTime
> 1 18m37s 16m18s –
> 48m7s 4m49s 41s{code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15975) Use LongAdder instead of AtomicLong
[
https://issues.apache.org/jira/browse/HDFS-15975?focusedWorklogId=584164&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584164
]
ASF GitHub Bot logged work on HDFS-15975:
-
Author: ASF GitHub Bot
Created on: 16/Apr/21 09:25
Start Date: 16/Apr/21 09:25
Worklog Time Spent: 10m
Work Description: tomscut commented on a change in pull request #2907:
URL: https://github.com/apache/hadoop/pull/2907#discussion_r614696047
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
##
@@ -607,11 +608,11 @@ public long getCacheCapacity() {
}
public long getNumBlocksFailedToCache() {
-return numBlocksFailedToCache.get();
+return numBlocksFailedToCache.longValue();
}
public long getNumBlocksFailedToUncache() {
-return numBlocksFailedToUncache.get();
+return numBlocksFailedToUncache.longValue();
}
public long getNumBlocksCached() {
Review comment:
Thank you @tasanuma for your careful review. I will be more careful when
I submit the code later. : )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 584164)
Time Spent: 2h 50m (was: 2h 40m)
> Use LongAdder instead of AtomicLong
> ---
>
> Key: HDFS-15975
> URL: https://issues.apache.org/jira/browse/HDFS-15975
> Project: Hadoop HDFS
> Issue Type: Wish
>Reporter: tomscut
>Assignee: tomscut
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> When counting some indicators, we can use LongAdder instead of AtomicLong to
> improve performance. The long value is not an atomic snapshot in LongAdder,
> but I think we can tolerate that.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15987) Improve oiv tool to parse fsimage file in parallel with delimited format
[
https://issues.apache.org/jira/browse/HDFS-15987?focusedWorklogId=584160&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584160
]
ASF GitHub Bot logged work on HDFS-15987:
-
Author: ASF GitHub Bot
Created on: 16/Apr/21 09:18
Start Date: 16/Apr/21 09:18
Worklog Time Spent: 10m
Work Description: whbing opened a new pull request #2918:
URL: https://github.com/apache/hadoop/pull/2918
…elimited format
## NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.)
For more details, please see
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 584160)
Remaining Estimate: 0h
Time Spent: 10m
> Improve oiv tool to parse fsimage file in parallel with delimited format
>
>
> Key: HDFS-15987
> URL: https://issues.apache.org/jira/browse/HDFS-15987
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Hongbing Wang
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The purpose of this Jira is to improve oiv tool to parse fsimage file with
> sub-sections (see -HDFS-14617-) in parallel with delmited format.
> 1.Serial parsing is time-consuming
> The time to serially parse a large fsimage with delimited format (e.g. `hdfs
> oiv -p Delimited -t ...`) is as follows:
> {code:java}
> 1) Loading string table: -> Not time consuming.
> 2) Loading inode references: -> Not time consuming
> 3) Loading directories in INode section: -> Slightly time consuming (3%)
> 4) Loading INode directory section: -> A bit time consuming (11%)
> 5) Output: -> Very time consuming (86%){code}
> Therefore, output is the most parallelized stage.
> 2.How to output in parallel
> The sub-sections are grouped in order, and each thread processes a group and
> outputs it to the file corresponding to each thread, and finally merges the
> output files.
> 3. The result of a test
> {code:java}
> input fsimage file info:
> 3.4G, 12 sub-sections, 55976500 INodes
> -
> Threads TotalTime OutputTime MergeTime
> 1 18m37s 16m18s –
> 48m7s 4m49s 41s{code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15975) Use LongAdder instead of AtomicLong
[
https://issues.apache.org/jira/browse/HDFS-15975?focusedWorklogId=584162&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584162
]
ASF GitHub Bot logged work on HDFS-15975:
-
Author: ASF GitHub Bot
Created on: 16/Apr/21 09:20
Start Date: 16/Apr/21 09:20
Worklog Time Spent: 10m
Work Description: tasanuma commented on a change in pull request #2907:
URL: https://github.com/apache/hadoop/pull/2907#discussion_r614693047
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
##
@@ -607,11 +608,11 @@ public long getCacheCapacity() {
}
public long getNumBlocksFailedToCache() {
-return numBlocksFailedToCache.get();
+return numBlocksFailedToCache.longValue();
}
public long getNumBlocksFailedToUncache() {
-return numBlocksFailedToUncache.get();
+return numBlocksFailedToUncache.longValue();
}
public long getNumBlocksCached() {
Review comment:
@tomscut Now we can remove `import
java.util.concurrent.atomic.AtomicLong` in FsDatasetCache.java. Others look
good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 584162)
Time Spent: 2h 40m (was: 2.5h)
> Use LongAdder instead of AtomicLong
> ---
>
> Key: HDFS-15975
> URL: https://issues.apache.org/jira/browse/HDFS-15975
> Project: Hadoop HDFS
> Issue Type: Wish
>Reporter: tomscut
>Assignee: tomscut
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> When counting some indicators, we can use LongAdder instead of AtomicLong to
> improve performance. The long value is not an atomic snapshot in LongAdder,
> but I think we can tolerate that.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15963) Unreleased volume references cause an infinite loop
[
https://issues.apache.org/jira/browse/HDFS-15963?focusedWorklogId=584161&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584161
]
ASF GitHub Bot logged work on HDFS-15963:
-
Author: ASF GitHub Bot
Created on: 16/Apr/21 09:18
Start Date: 16/Apr/21 09:18
Worklog Time Spent: 10m
Work Description: Hexiaoqiao commented on pull request #2889:
URL: https://github.com/apache/hadoop/pull/2889#issuecomment-821042143
@jojochuang Hi Wei-Chiu, any furthermore comment here? If not, I would like
to commit this PR shortly. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 584161)
Time Spent: 3.5h (was: 3h 20m)
> Unreleased volume references cause an infinite loop
> ---
>
> Key: HDFS-15963
> URL: https://issues.apache.org/jira/browse/HDFS-15963
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Shuyan Zhang
>Assignee: Shuyan Zhang
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-15963.001.patch, HDFS-15963.002.patch,
> HDFS-15963.003.patch
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> When BlockSender throws an exception because the meta-data cannot be found,
> the volume reference obtained by the thread is not released, which causes the
> thread trying to remove the volume to wait and fall into an infinite loop.
> {code:java}
> boolean checkVolumesRemoved() {
> Iterator it = volumesBeingRemoved.iterator();
> while (it.hasNext()) {
> FsVolumeImpl volume = it.next();
> if (!volume.checkClosed()) {
> return false;
> }
> it.remove();
> }
> return true;
> }
> boolean checkClosed() {
> // always be true.
> if (this.reference.getReferenceCount() > 0) {
> FsDatasetImpl.LOG.debug("The reference count for {} is {}, wait to be 0.",
> this, reference.getReferenceCount());
> return false;
> }
> return true;
> }
> {code}
> At the same time, because the thread has been holding checkDirsLock when
> removing the volume, other threads trying to acquire the same lock will be
> permanently blocked.
> Similar problems also occur in RamDiskAsyncLazyPersistService and
> FsDatasetAsyncDiskService.
> This patch releases the three previously unreleased volume references.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15987) Improve oiv tool to parse fsimage file in parallel with delimited format
[
https://issues.apache.org/jira/browse/HDFS-15987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-15987:
--
Labels: pull-request-available (was: )
> Improve oiv tool to parse fsimage file in parallel with delimited format
>
>
> Key: HDFS-15987
> URL: https://issues.apache.org/jira/browse/HDFS-15987
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Hongbing Wang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The purpose of this Jira is to improve oiv tool to parse fsimage file with
> sub-sections (see -HDFS-14617-) in parallel with delmited format.
> 1.Serial parsing is time-consuming
> The time to serially parse a large fsimage with delimited format (e.g. `hdfs
> oiv -p Delimited -t ...`) is as follows:
> {code:java}
> 1) Loading string table: -> Not time consuming.
> 2) Loading inode references: -> Not time consuming
> 3) Loading directories in INode section: -> Slightly time consuming (3%)
> 4) Loading INode directory section: -> A bit time consuming (11%)
> 5) Output: -> Very time consuming (86%){code}
> Therefore, output is the most parallelized stage.
> 2.How to output in parallel
> The sub-sections are grouped in order, and each thread processes a group and
> outputs it to the file corresponding to each thread, and finally merges the
> output files.
> 3. The result of a test
> {code:java}
> input fsimage file info:
> 3.4G, 12 sub-sections, 55976500 INodes
> -
> Threads TotalTime OutputTime MergeTime
> 1 18m37s 16m18s –
> 48m7s 4m49s 41s{code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15984) Deleted data on the Web UI must be saved to the trash
[ https://issues.apache.org/jira/browse/HDFS-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HDFS-15984. --- Resolution: Duplicate > Deleted data on the Web UI must be saved to the trash > -- > > Key: HDFS-15984 > URL: https://issues.apache.org/jira/browse/HDFS-15984 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs >Reporter: Bhavik Patel >Priority: Major > > If we delete the data from the Web UI then it should be first moved to > configured/default Trash directory and after the trash interval time, it > should be removed. currently, data directly removed from the system[This > behavior should be the same as CLI cmd] > > This can be helpful when the user accidentally deletes data from the Web UI. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15975) Use LongAdder instead of AtomicLong
[
https://issues.apache.org/jira/browse/HDFS-15975?focusedWorklogId=584159&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584159
]
ASF GitHub Bot logged work on HDFS-15975:
-
Author: ASF GitHub Bot
Created on: 16/Apr/21 09:14
Start Date: 16/Apr/21 09:14
Worklog Time Spent: 10m
Work Description: tomscut commented on a change in pull request #2907:
URL: https://github.com/apache/hadoop/pull/2907#discussion_r614688538
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
##
@@ -607,11 +608,11 @@ public long getCacheCapacity() {
}
public long getNumBlocksFailedToCache() {
-return numBlocksFailedToCache.get();
+return numBlocksFailedToCache.longValue();
}
public long getNumBlocksFailedToUncache() {
-return numBlocksFailedToUncache.get();
+return numBlocksFailedToUncache.longValue();
}
public long getNumBlocksCached() {
Review comment:
I fixed that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 584159)
Time Spent: 2.5h (was: 2h 20m)
> Use LongAdder instead of AtomicLong
> ---
>
> Key: HDFS-15975
> URL: https://issues.apache.org/jira/browse/HDFS-15975
> Project: Hadoop HDFS
> Issue Type: Wish
>Reporter: tomscut
>Assignee: tomscut
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> When counting some indicators, we can use LongAdder instead of AtomicLong to
> improve performance. The long value is not an atomic snapshot in LongAdder,
> but I think we can tolerate that.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15975) Use LongAdder instead of AtomicLong
[
https://issues.apache.org/jira/browse/HDFS-15975?focusedWorklogId=584157&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584157
]
ASF GitHub Bot logged work on HDFS-15975:
-
Author: ASF GitHub Bot
Created on: 16/Apr/21 09:10
Start Date: 16/Apr/21 09:10
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2907:
URL: https://github.com/apache/hadoop/pull/2907#issuecomment-821037653
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 28s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 21s | | trunk passed |
| +1 :green_heart: | compile | 20m 38s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 17m 58s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 3m 46s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 15s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 5s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 4m 9s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 8m 13s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 30s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 55s | | the patch passed |
| +1 :green_heart: | compile | 20m 3s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 20m 3s | | the patch passed |
| +1 :green_heart: | compile | 18m 2s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 18m 2s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 42s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2907/3/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 6 new + 242 unchanged - 5 fixed = 248 total (was
247) |
| +1 :green_heart: | mvnsite | 4m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 0s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 4m 6s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 8m 43s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 35s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 23s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 35s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 235m 13s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2907/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 19s | | The patch does not
generate ASF License warnings. |
| | | 453m 10s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2907/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2907 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite u
[jira] [Resolved] (HDFS-15983) Deleted data on the Web UI must be saved to the trash
[ https://issues.apache.org/jira/browse/HDFS-15983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HDFS-15983. --- Resolution: Duplicate > Deleted data on the Web UI must be saved to the trash > -- > > Key: HDFS-15983 > URL: https://issues.apache.org/jira/browse/HDFS-15983 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs >Reporter: Bhavik Patel >Priority: Major > > If we delete the data from the Web UI then it should be first moved to > configured/default Trash directory and after the trash interval time, it > should be removed. currently, data directly removed from the system[This > behavior should be the same as CLI cmd] > > This can be helpful when the user accidentally deletes data from the Web UI. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15986) Restrict the "-skipTrash" param for accidentally deletes data
[ https://issues.apache.org/jira/browse/HDFS-15986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17322734#comment-17322734 ] Steve Loughran commented on HDFS-15986: --- well, I can see what you've just done :) I think changes to the CLI is going to be trouble as there's some guarantees of consistency over time, with command line, behaviour and of output. hadoop fs -rm -skipTrash is used in scripts, so we can't suddenly say "this will be prompted". And if we add an override "-f" then people will just get into the habit of going "hadoop fs -rm -R -f -skipTrash" the way "rm -rf" is my standard rm string on the command line. > Restrict the "-skipTrash" param for accidentally deletes data > - > > Key: HDFS-15986 > URL: https://issues.apache.org/jira/browse/HDFS-15986 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Bhavik Patel >Priority: Major > > Suppose the user tries to delete the data from CLI with the "-skipTrash" > param but by mistake, he deleted a couple of directories but actually, that > directory user want to retain then their is no way to retrieve the delete > data. > It will be good to have a confirm message like: "Skip the trash for the > hdfs:///dri1/file.txt files? (Y or N)" ro we can completely disable the > "-skipTrash" param. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15987) Improve oiv tool to parse fsimage file in parallel with delimited format
Hongbing Wang created HDFS-15987:
Summary: Improve oiv tool to parse fsimage file in parallel with
delimited format
Key: HDFS-15987
URL: https://issues.apache.org/jira/browse/HDFS-15987
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Hongbing Wang
The purpose of this Jira is to improve oiv tool to parse fsimage file with
sub-sections (see -HDFS-14617-) in parallel with delmited format.
1.Serial parsing is time-consuming
The time to serially parse a large fsimage with delimited format (e.g. `hdfs
oiv -p Delimited -t ...`) is as follows:
{code:java}
1) Loading string table: -> Not time consuming.
2) Loading inode references: -> Not time consuming
3) Loading directories in INode section: -> Slightly time consuming (3%)
4) Loading INode directory section: -> A bit time consuming (11%)
5) Output: -> Very time consuming (86%){code}
Therefore, output is the most parallelized stage.
2.How to output in parallel
The sub-sections are grouped in order, and each thread processes a group and
outputs it to the file corresponding to each thread, and finally merges the
output files.
3. The result of a test
{code:java}
input fsimage file info:
3.4G, 12 sub-sections, 55976500 INodes
-
Threads TotalTime OutputTime MergeTime
1 18m37s 16m18s –
48m7s 4m49s 41s{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14698) Upgrade OfflineImageViewer to be compatible with new FSImage format
[ https://issues.apache.org/jira/browse/HDFS-14698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17322722#comment-17322722 ] JiangHua Zhu commented on HDFS-14698: - [~hexiaoqiao], I am very interested in this question. Can I solve it? > Upgrade OfflineImageViewer to be compatible with new FSImage format > --- > > Key: HDFS-14698 > URL: https://issues.apache.org/jira/browse/HDFS-14698 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Xiaoqiao He >Priority: Major > > OIV tools upgrade to be compatible with new FSImage format using by parallel > loading. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15967) Improve the log for Short Circuit Local Reads
[ https://issues.apache.org/jira/browse/HDFS-15967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17322719#comment-17322719 ] Akira Ajisaka commented on HDFS-15967: -- Apache id is required to log in to Jenkins and trigger a job manually. You can trigger a job by changing the status to open and then change to patch available. > Improve the log for Short Circuit Local Reads > - > > Key: HDFS-15967 > URL: https://issues.apache.org/jira/browse/HDFS-15967 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Bhavik Patel >Assignee: Bhavik Patel >Priority: Minor > Attachments: HDFS-15967.001.patch > > > Improve the log for Short Circuit Local Reads -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15934) Make DirectoryScanner reconcile blocks batch size and interval between batch configurable.
[
https://issues.apache.org/jira/browse/HDFS-15934?focusedWorklogId=584143&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584143
]
ASF GitHub Bot logged work on HDFS-15934:
-
Author: ASF GitHub Bot
Created on: 16/Apr/21 08:25
Start Date: 16/Apr/21 08:25
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2833:
URL: https://github.com/apache/hadoop/pull/2833#issuecomment-821010516
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 54s | | trunk passed |
| +1 :green_heart: | compile | 1m 19s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 21s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 32s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 16s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 57s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 17s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 44s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 18s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 237m 0s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 55s | | The patch does not
generate ASF License warnings. |
| | | 325m 37s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2833 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 4b1137f6cfbd 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/ha
[jira] [Created] (HDFS-15986) Restrict the "-skipTrash" param for accidentally deletes data
Bhavik Patel created HDFS-15986: --- Summary: Restrict the "-skipTrash" param for accidentally deletes data Key: HDFS-15986 URL: https://issues.apache.org/jira/browse/HDFS-15986 Project: Hadoop HDFS Issue Type: Improvement Reporter: Bhavik Patel Suppose the user tries to delete the data from CLI with the "-skipTrash" param but by mistake, he deleted a couple of directories but actually, that directory user want to retain then their is no way to retrieve the delete data. It will be good to have a confirm message like: "Skip the trash for the hdfs:///dri1/file.txt files? (Y or N)" ro we can completely disable the "-skipTrash" param. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15985) Incorrect sorting will cause failure to load an FsImage file
[
https://issues.apache.org/jira/browse/HDFS-15985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17322701#comment-17322701
]
Wei-Chiu Chuang commented on HDFS-15985:
[~sodonnell]
> Incorrect sorting will cause failure to load an FsImage file
>
>
> Key: HDFS-15985
> URL: https://issues.apache.org/jira/browse/HDFS-15985
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> After we have introduced HDFS-14617 or HDFS-14771, when loading an fsimage
> file, the following error will pop up:
> 2021-04-15 17:21:17,868 [293072]-INFO [main:FSImage@784]-Planning to load
> image:
> FSImageFile(file=//hadoop/hdfs/namenode/current/fsimage_0,
> cpktTxId=0)
> 2021-04-15 17:25:53,288 [568492]-INFO
> [main:FSImageFormatPBINode$Loader@229]-Loading 725097952 INodes.
> 2021-04-15 17:25:53,289 [568493]-ERROR [main:FSImage@730]-Failed to load
> image from
> FSImageFile(file=//hadoop/hdfs/namenode/current/fsimage_0,
> cpktTxId=0)
> java.lang.IllegalStateException: GLOBAL: serial number 3 does not exist
> at
> org.apache.hadoop.hdfs.server.namenode.SerialNumberMap.get(SerialNumberMap.java:85)
> at
> org.apache.hadoop.hdfs.server.namenode.SerialNumberManager.getString(SerialNumberManager.java:121)
> at
> org.apache.hadoop.hdfs.server.namenode.SerialNumberManager.getString(SerialNumberManager.java:125)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeWithAdditionalFields$PermissionStatusFormat.toPermissionStatus(INodeWithAdditionalFields.java:86)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadPermission(FSImageFormatPBINode.java:93)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeFile(FSImageFormatPBINode.java:303)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINode(FSImageFormatPBINode.java:280)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeSection(FSImageFormatPBINode.java:237)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:237)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:176)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:226)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:937)
> It was found that this anomaly was related to sorting, as follows:
> ArrayList sections = Lists.newArrayList(summary
> .getSectionsList());
> Collections.sort(sections, new Comparator() {
> @Override
> public int compare(FileSummary.Section s1, FileSummary.Section s2) {
> SectionName n1 = SectionName.fromString(s1.getName());
> SectionName n2 = SectionName.fromString(s2.getName());
> if (n1 == null) {
> return n2 == null? 0: -1;
> } else if (n2 == null) {
> return -1;
> } else {
> return n1.ordinal()-n2.ordinal();
> }
> }
> });
> When n1 != null and n2 == null, this will cause sorting errors.
> When loading Sections, the correct order of loading Sections:
> NS_INFO -> STRING_TABLE -> INODE
> If the sorting is incorrect, the loading order is as follows:
> INDOE -> NS_INFO -> STRING_TABLE
> Because when loading INODE, you need to rely on STRING_TABLE.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-15985) Incorrect sorting will cause failure to load an FsImage file
[
https://issues.apache.org/jira/browse/HDFS-15985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-15985 started by JiangHua Zhu.
---
> Incorrect sorting will cause failure to load an FsImage file
>
>
> Key: HDFS-15985
> URL: https://issues.apache.org/jira/browse/HDFS-15985
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> After we have introduced HDFS-14617 or HDFS-14771, when loading an fsimage
> file, the following error will pop up:
> 2021-04-15 17:21:17,868 [293072]-INFO [main:FSImage@784]-Planning to load
> image:
> FSImageFile(file=//hadoop/hdfs/namenode/current/fsimage_0,
> cpktTxId=0)
> 2021-04-15 17:25:53,288 [568492]-INFO
> [main:FSImageFormatPBINode$Loader@229]-Loading 725097952 INodes.
> 2021-04-15 17:25:53,289 [568493]-ERROR [main:FSImage@730]-Failed to load
> image from
> FSImageFile(file=//hadoop/hdfs/namenode/current/fsimage_0,
> cpktTxId=0)
> java.lang.IllegalStateException: GLOBAL: serial number 3 does not exist
> at
> org.apache.hadoop.hdfs.server.namenode.SerialNumberMap.get(SerialNumberMap.java:85)
> at
> org.apache.hadoop.hdfs.server.namenode.SerialNumberManager.getString(SerialNumberManager.java:121)
> at
> org.apache.hadoop.hdfs.server.namenode.SerialNumberManager.getString(SerialNumberManager.java:125)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeWithAdditionalFields$PermissionStatusFormat.toPermissionStatus(INodeWithAdditionalFields.java:86)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadPermission(FSImageFormatPBINode.java:93)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeFile(FSImageFormatPBINode.java:303)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINode(FSImageFormatPBINode.java:280)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeSection(FSImageFormatPBINode.java:237)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:237)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:176)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:226)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:937)
> It was found that this anomaly was related to sorting, as follows:
> ArrayList sections = Lists.newArrayList(summary
> .getSectionsList());
> Collections.sort(sections, new Comparator() {
> @Override
> public int compare(FileSummary.Section s1, FileSummary.Section s2) {
> SectionName n1 = SectionName.fromString(s1.getName());
> SectionName n2 = SectionName.fromString(s2.getName());
> if (n1 == null) {
> return n2 == null? 0: -1;
> } else if (n2 == null) {
> return -1;
> } else {
> return n1.ordinal()-n2.ordinal();
> }
> }
> });
> When n1 != null and n2 == null, this will cause sorting errors.
> When loading Sections, the correct order of loading Sections:
> NS_INFO -> STRING_TABLE -> INODE
> If the sorting is incorrect, the loading order is as follows:
> INDOE -> NS_INFO -> STRING_TABLE
> Because when loading INODE, you need to rely on STRING_TABLE.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15756) RBF: Cannot get updated delegation token from zookeeper
[ https://issues.apache.org/jira/browse/HDFS-15756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17322696#comment-17322696 ] zhangxiping commented on HDFS-15756: [~John Smith] Wow , looking at the attached results in Hadoop-16828, the improvement is obvious.Currently Token is created by Router and Token's Seq is realized by ZK's Sharecount and shared among Router.We plan to join our cluster, thank you very much! > RBF: Cannot get updated delegation token from zookeeper > --- > > Key: HDFS-15756 > URL: https://issues.apache.org/jira/browse/HDFS-15756 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.0.0 >Reporter: hbprotoss >Priority: Major > > Affected version: all version with rbf > When RBF work with spark 2.4 client mode, there will be a chance that token > is missing across different nodes in RBF cluster. The root cause is that > spark renew the token(via resource manager) immediately after got one, as > zookeeper don't have a strong consistency guarantee after an update in > cluster, zookeeper client may read a stale value in some followers not synced > with other nodes. > > We apply a patch in spark, but it is still the problem of RBF. Is it possible > for RBF to replace the delegation token store using some other > datasource(redis for example)? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15985) Incorrect sorting will cause failure to load an FsImage file
[
https://issues.apache.org/jira/browse/HDFS-15985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-15985:
--
Labels: pull-request-available (was: )
> Incorrect sorting will cause failure to load an FsImage file
>
>
> Key: HDFS-15985
> URL: https://issues.apache.org/jira/browse/HDFS-15985
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> After we have introduced HDFS-14617 or HDFS-14771, when loading an fsimage
> file, the following error will pop up:
> 2021-04-15 17:21:17,868 [293072]-INFO [main:FSImage@784]-Planning to load
> image:
> FSImageFile(file=//hadoop/hdfs/namenode/current/fsimage_0,
> cpktTxId=0)
> 2021-04-15 17:25:53,288 [568492]-INFO
> [main:FSImageFormatPBINode$Loader@229]-Loading 725097952 INodes.
> 2021-04-15 17:25:53,289 [568493]-ERROR [main:FSImage@730]-Failed to load
> image from
> FSImageFile(file=//hadoop/hdfs/namenode/current/fsimage_0,
> cpktTxId=0)
> java.lang.IllegalStateException: GLOBAL: serial number 3 does not exist
> at
> org.apache.hadoop.hdfs.server.namenode.SerialNumberMap.get(SerialNumberMap.java:85)
> at
> org.apache.hadoop.hdfs.server.namenode.SerialNumberManager.getString(SerialNumberManager.java:121)
> at
> org.apache.hadoop.hdfs.server.namenode.SerialNumberManager.getString(SerialNumberManager.java:125)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeWithAdditionalFields$PermissionStatusFormat.toPermissionStatus(INodeWithAdditionalFields.java:86)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadPermission(FSImageFormatPBINode.java:93)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeFile(FSImageFormatPBINode.java:303)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINode(FSImageFormatPBINode.java:280)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeSection(FSImageFormatPBINode.java:237)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:237)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:176)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:226)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:937)
> It was found that this anomaly was related to sorting, as follows:
> ArrayList sections = Lists.newArrayList(summary
> .getSectionsList());
> Collections.sort(sections, new Comparator() {
> @Override
> public int compare(FileSummary.Section s1, FileSummary.Section s2) {
> SectionName n1 = SectionName.fromString(s1.getName());
> SectionName n2 = SectionName.fromString(s2.getName());
> if (n1 == null) {
> return n2 == null? 0: -1;
> } else if (n2 == null) {
> return -1;
> } else {
> return n1.ordinal()-n2.ordinal();
> }
> }
> });
> When n1 != null and n2 == null, this will cause sorting errors.
> When loading Sections, the correct order of loading Sections:
> NS_INFO -> STRING_TABLE -> INODE
> If the sorting is incorrect, the loading order is as follows:
> INDOE -> NS_INFO -> STRING_TABLE
> Because when loading INODE, you need to rely on STRING_TABLE.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15985) Incorrect sorting will cause failure to load an FsImage file
[
https://issues.apache.org/jira/browse/HDFS-15985?focusedWorklogId=584123&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584123
]
ASF GitHub Bot logged work on HDFS-15985:
-
Author: ASF GitHub Bot
Created on: 16/Apr/21 07:49
Start Date: 16/Apr/21 07:49
Worklog Time Spent: 10m
Work Description: jianghuazhu opened a new pull request #2917:
URL: https://github.com/apache/hadoop/pull/2917
## NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.)
For more details, please see
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 584123)
Remaining Estimate: 0h
Time Spent: 10m
> Incorrect sorting will cause failure to load an FsImage file
>
>
> Key: HDFS-15985
> URL: https://issues.apache.org/jira/browse/HDFS-15985
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> After we have introduced HDFS-14617 or HDFS-14771, when loading an fsimage
> file, the following error will pop up:
> 2021-04-15 17:21:17,868 [293072]-INFO [main:FSImage@784]-Planning to load
> image:
> FSImageFile(file=//hadoop/hdfs/namenode/current/fsimage_0,
> cpktTxId=0)
> 2021-04-15 17:25:53,288 [568492]-INFO
> [main:FSImageFormatPBINode$Loader@229]-Loading 725097952 INodes.
> 2021-04-15 17:25:53,289 [568493]-ERROR [main:FSImage@730]-Failed to load
> image from
> FSImageFile(file=//hadoop/hdfs/namenode/current/fsimage_0,
> cpktTxId=0)
> java.lang.IllegalStateException: GLOBAL: serial number 3 does not exist
> at
> org.apache.hadoop.hdfs.server.namenode.SerialNumberMap.get(SerialNumberMap.java:85)
> at
> org.apache.hadoop.hdfs.server.namenode.SerialNumberManager.getString(SerialNumberManager.java:121)
> at
> org.apache.hadoop.hdfs.server.namenode.SerialNumberManager.getString(SerialNumberManager.java:125)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeWithAdditionalFields$PermissionStatusFormat.toPermissionStatus(INodeWithAdditionalFields.java:86)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadPermission(FSImageFormatPBINode.java:93)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeFile(FSImageFormatPBINode.java:303)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINode(FSImageFormatPBINode.java:280)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeSection(FSImageFormatPBINode.java:237)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:237)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:176)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:226)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:937)
> It was found that this anomaly was related to sorting, as follows:
> ArrayList sections = Lists.newArrayList(summary
> .getSectionsList());
> Collections.sort(sections, new Comparator() {
> @Override
> public int compare(FileSummary.Section s1, FileSummary.Section s2) {
> SectionName n1 = SectionName.fromString(s1.getName());
> SectionName n2 = SectionName.fromString(s2.getName());
> if (n1 == null) {
> return n2 == null? 0: -1;
> } else if (n2 == null) {
> return -1;
> } else {
> return n1.ordinal()-n2.ordinal();
> }
> }
> });
> When n1 != null and n2 == null, this will cause sorting errors.
> When loading Sections, the correct order of loading Sections:
> NS_INFO -> STRING_TABLE -> INODE
> If the sorting is incorrect, the loading order is as follows:
> INDOE -> NS_INFO -> STRING_TABLE
> Because when loading INODE, you need to rely on STRING_TABLE.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands,
[jira] [Assigned] (HDFS-15985) Incorrect sorting will cause failure to load an FsImage file
[
https://issues.apache.org/jira/browse/HDFS-15985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
JiangHua Zhu reassigned HDFS-15985:
---
Assignee: JiangHua Zhu
> Incorrect sorting will cause failure to load an FsImage file
>
>
> Key: HDFS-15985
> URL: https://issues.apache.org/jira/browse/HDFS-15985
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
>
> After we have introduced HDFS-14617 or HDFS-14771, when loading an fsimage
> file, the following error will pop up:
> 2021-04-15 17:21:17,868 [293072]-INFO [main:FSImage@784]-Planning to load
> image:
> FSImageFile(file=//hadoop/hdfs/namenode/current/fsimage_0,
> cpktTxId=0)
> 2021-04-15 17:25:53,288 [568492]-INFO
> [main:FSImageFormatPBINode$Loader@229]-Loading 725097952 INodes.
> 2021-04-15 17:25:53,289 [568493]-ERROR [main:FSImage@730]-Failed to load
> image from
> FSImageFile(file=//hadoop/hdfs/namenode/current/fsimage_0,
> cpktTxId=0)
> java.lang.IllegalStateException: GLOBAL: serial number 3 does not exist
> at
> org.apache.hadoop.hdfs.server.namenode.SerialNumberMap.get(SerialNumberMap.java:85)
> at
> org.apache.hadoop.hdfs.server.namenode.SerialNumberManager.getString(SerialNumberManager.java:121)
> at
> org.apache.hadoop.hdfs.server.namenode.SerialNumberManager.getString(SerialNumberManager.java:125)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeWithAdditionalFields$PermissionStatusFormat.toPermissionStatus(INodeWithAdditionalFields.java:86)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadPermission(FSImageFormatPBINode.java:93)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeFile(FSImageFormatPBINode.java:303)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINode(FSImageFormatPBINode.java:280)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeSection(FSImageFormatPBINode.java:237)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:237)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:176)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:226)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:937)
> It was found that this anomaly was related to sorting, as follows:
> ArrayList sections = Lists.newArrayList(summary
> .getSectionsList());
> Collections.sort(sections, new Comparator() {
> @Override
> public int compare(FileSummary.Section s1, FileSummary.Section s2) {
> SectionName n1 = SectionName.fromString(s1.getName());
> SectionName n2 = SectionName.fromString(s2.getName());
> if (n1 == null) {
> return n2 == null? 0: -1;
> } else if (n2 == null) {
> return -1;
> } else {
> return n1.ordinal()-n2.ordinal();
> }
> }
> });
> When n1 != null and n2 == null, this will cause sorting errors.
> When loading Sections, the correct order of loading Sections:
> NS_INFO -> STRING_TABLE -> INODE
> If the sorting is incorrect, the loading order is as follows:
> INDOE -> NS_INFO -> STRING_TABLE
> Because when loading INODE, you need to rely on STRING_TABLE.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15985) Incorrect sorting will cause failure to load an FsImage file
JiangHua Zhu created HDFS-15985:
---
Summary: Incorrect sorting will cause failure to load an FsImage
file
Key: HDFS-15985
URL: https://issues.apache.org/jira/browse/HDFS-15985
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: JiangHua Zhu
After we have introduced HDFS-14617 or HDFS-14771, when loading an fsimage
file, the following error will pop up:
2021-04-15 17:21:17,868 [293072]-INFO [main:FSImage@784]-Planning to load
image:
FSImageFile(file=//hadoop/hdfs/namenode/current/fsimage_0,
cpktTxId=0)
2021-04-15 17:25:53,288 [568492]-INFO
[main:FSImageFormatPBINode$Loader@229]-Loading 725097952 INodes.
2021-04-15 17:25:53,289 [568493]-ERROR [main:FSImage@730]-Failed to load image
from FSImageFile(file=//hadoop/hdfs/namenode/current/fsimage_0,
cpktTxId=0)
java.lang.IllegalStateException: GLOBAL: serial number 3 does not exist
at
org.apache.hadoop.hdfs.server.namenode.SerialNumberMap.get(SerialNumberMap.java:85)
at
org.apache.hadoop.hdfs.server.namenode.SerialNumberManager.getString(SerialNumberManager.java:121)
at
org.apache.hadoop.hdfs.server.namenode.SerialNumberManager.getString(SerialNumberManager.java:125)
at
org.apache.hadoop.hdfs.server.namenode.INodeWithAdditionalFields$PermissionStatusFormat.toPermissionStatus(INodeWithAdditionalFields.java:86)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadPermission(FSImageFormatPBINode.java:93)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeFile(FSImageFormatPBINode.java:303)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINode(FSImageFormatPBINode.java:280)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeSection(FSImageFormatPBINode.java:237)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:237)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:176)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:226)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:937)
It was found that this anomaly was related to sorting, as follows:
ArrayList sections = Lists.newArrayList(summary
.getSectionsList());
Collections.sort(sections, new Comparator() {
@Override
public int compare(FileSummary.Section s1, FileSummary.Section s2) {
SectionName n1 = SectionName.fromString(s1.getName());
SectionName n2 = SectionName.fromString(s2.getName());
if (n1 == null) {
return n2 == null? 0: -1;
} else if (n2 == null) {
return -1;
} else {
return n1.ordinal()-n2.ordinal();
}
}
});
When n1 != null and n2 == null, this will cause sorting errors.
When loading Sections, the correct order of loading Sections:
NS_INFO -> STRING_TABLE -> INODE
If the sorting is incorrect, the loading order is as follows:
INDOE -> NS_INFO -> STRING_TABLE
Because when loading INODE, you need to rely on STRING_TABLE.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14263) Remove unnecessary block file exists check from FsDatasetImpl#getBlockInputStream()
[
https://issues.apache.org/jira/browse/HDFS-14263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shuyan Zhang updated HDFS-14263:
Description: As discussed in HDFS-10636,
{{FsDatasetImpl#getBlockInputStream()}} doing unnecessary block replica exist
check. (was: * As discussed in HDFS-10636,
{{FsDatasetImpl#getBlockInputStream()}} doing unnecessary block replica exist
check.)
> Remove unnecessary block file exists check from
> FsDatasetImpl#getBlockInputStream()
> ---
>
> Key: HDFS-14263
> URL: https://issues.apache.org/jira/browse/HDFS-14263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14263.001.patch, HDFS-14263.002.patch
>
>
> As discussed in HDFS-10636, {{FsDatasetImpl#getBlockInputStream()}} doing
> unnecessary block replica exist check.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14263) Remove unnecessary block file exists check from FsDatasetImpl#getBlockInputStream()
[
https://issues.apache.org/jira/browse/HDFS-14263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shuyan Zhang updated HDFS-14263:
Description: * As discussed in HDFS-10636,
{{FsDatasetImpl#getBlockInputStream()}} doing unnecessary block replica exist
check. (was: As discussed in HDFS-10636,
{{FsDatasetImpl#getBlockInputStream()}} doing unnecessary block replica exist
check.)
> Remove unnecessary block file exists check from
> FsDatasetImpl#getBlockInputStream()
> ---
>
> Key: HDFS-14263
> URL: https://issues.apache.org/jira/browse/HDFS-14263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14263.001.patch, HDFS-14263.002.patch
>
>
> * As discussed in HDFS-10636, {{FsDatasetImpl#getBlockInputStream()}} doing
> unnecessary block replica exist check.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15756) RBF: Cannot get updated delegation token from zookeeper
[
https://issues.apache.org/jira/browse/HDFS-15756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17322640#comment-17322640
]
zhangxiping edited comment on HDFS-15756 at 4/16/21, 7:05 AM:
--
[~hexiaoqiao] Thank you for your reply. Yes, I have the same question. I
think Spark should not renew token so soon.To avoid the current problem, we
implement that the Router waits for 100ms to retry to get the token(recently
generated) from ZK.Since the probability of this problem is relatively small,
it occurs dozens of times a day and does not affect the performance of the
Router, it can also be obtained from ZK.But I think it would be better to try
again on the client side, which is not a perfect solution and can only
circumvent the problem temporarily. Expect a better solution.
was (Author: zhangxiping):
[~hexiaoqiao] Thank you for your reply. Yes, I have the same question. I
think Spark should not renew token so soon.To avoid the current problem, we
implement that the Router waits for 100ms to retry to get the token(recently
generated) from ZK.Since the probability of this problem is relatively small,
it occurs dozens of times a day and does not affect the performance of the
Router, it can also be obtained from ZK.But I think it would be better to try
again on the client side, which is not a perfect solution and can only
circumvent the problem temporarily. Expect a better solution.
{code:java}
//代码占位符 AbstractDelegationTokenSecretManager
protected synchronized byte[] createPassword(TokenIdent identifier) {
int sequenceNum;
long now = Time.now();
sequenceNum = incrementDelegationTokenSeqNum();
identifier.setIssueDate(now);
identifier.setMaxDate(now + tokenMaxLifetime);
identifier.setMasterKeyId(currentKey.getKeyId());
identifier.setSequenceNumber(sequenceNum);
LOG.info("Creating password for identifier: " + formatTokenId(identifier)
+ ", currentKey: " + currentKey.getKeyId());
byte[] password = createPassword(identifier.getBytes(), currentKey.getKey());
DelegationTokenInformation tokenInfo = new DelegationTokenInformation(now
+ tokenRenewInterval, password, getTrackingIdIfEnabled(identifier));
try {
storeToken(identifier, tokenInfo);
} catch (IOException ioe) {
LOG.error("Could not store token " + formatTokenId(identifier) + "!!",
ioe);
}
return password;
}
{code}
> RBF: Cannot get updated delegation token from zookeeper
> ---
>
> Key: HDFS-15756
> URL: https://issues.apache.org/jira/browse/HDFS-15756
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Affects Versions: 3.0.0
>Reporter: hbprotoss
>Priority: Major
>
> Affected version: all version with rbf
> When RBF work with spark 2.4 client mode, there will be a chance that token
> is missing across different nodes in RBF cluster. The root cause is that
> spark renew the token(via resource manager) immediately after got one, as
> zookeeper don't have a strong consistency guarantee after an update in
> cluster, zookeeper client may read a stale value in some followers not synced
> with other nodes.
>
> We apply a patch in spark, but it is still the problem of RBF. Is it possible
> for RBF to replace the delegation token store using some other
> datasource(redis for example)?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
