[jira] [Comment Edited] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index
[
https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877508#comment-16877508
]
He Xiaoqiao edited comment on HDFS-14617 at 7/3/19 5:56 AM:
Just took a look, thanks a lot for this great work, it is very sensible. I have
verified this patch, and it works well mostly,
1. check compatibility with old fsimage format,
* new version fsimage loader can load old format correctly, of course without
parallel loading since there are no INODE_SUB or INODE_DIR_SUB or other *_SUB
sections. the following fsimage saving will go with new format then parallel
loading if restart.
* old version fsimage loader can not load new format fsimage (as expected)
since it could not parse new section.
2. not benchmark with large fsimage file, will do it later.
some comments but all fairly minor implementation or style details.
1. Agree to disable parallel loading if config CompressionCodec for fsimage as
TODO comments;
2. Some metrics or more log for loading parallel section time cost will be
helpful to get this improvement.
3. Lambda expressions suggestions, for instance:
FSImageFormatPBINode#loadINodeDirectorySectionInParallel L226 could replace by
lambda,
{code:java}
for (FileSummary.Section s : sections) {
service.submit(() -> {
..
}
}
{code}
4. Need stricter inspect about INode numbers, INodeDirectorySection.DirEntry
number, etc when using parallel loading?
Thanks again for the great work.
was (Author: hexiaoqiao):
Just took a look, thanks a lot for this great work. It is very sensible. I
verify this patch, and mostly it works well,
1. check compatibility with old fsimage format,
* new version fsimage loader can load old format correctly, of course without
parallel loading since there are no INODE_SUB or INODE_DIR_SUB or other *_SUB
sections. the following fsimage saving will go with new format then parallel
loading if restart.
* old version fsimage loader can not load new format fsimage (as expected)
since it could not parse new section.
2. not benchmark with large fsimage file, will do it later.
some comments but all fairly minor implementation or style details.
1. Agree to disable parallel loading if config CompressionCodec for fsimage as
TODO comments;
2. Some metrics or more log for loading parallel section time cost will be
helpful to get this improvement.
3. Lambda expressions suggestions, for instance:
FSImageFormatPBINode#loadINodeDirectorySectionInParallel L226 could replace by
lambda,
{code:java}
for (FileSummary.Section s : sections) {
service.submit(() -> {
..
}
}
{code}
4. Need stricter inspect about INode numbers, INodeDirectorySection.DirEntry
number, etc when using parallel loading?
Thanks again for the great work.
> Improve fsimage load time by writing sub-sections to the fsimage index
> --
>
> Key: HDFS-14617
> URL: https://issues.apache.org/jira/browse/HDFS-14617
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Attachments: HDFS-14617.001.patch
>
>
> Loading an fsimage is basically a single threaded process. The current
> fsimage is written out in sections, eg iNode, iNode_Directory, Snapshots,
> Snapshot_Diff etc. Then at the end of the file, an index is written that
> contains the offset and length of each section. The image loader code uses
> this index to initialize an input stream to read and process each section. It
> is important that one section is fully loaded before another is started, as
> the next section depends on the results of the previous one.
> What I would like to propose is the following:
> 1. When writing the image, we can optionally output sub_sections to the
> index. That way, a given section would effectively be split into several
> sections, eg:
> {code:java}
>inode_section offset 10 length 1000
> inode_sub_section offset 10 length 500
> inode_sub_section offset 510 length 500
>
>inode_dir_section offset 1010 length 1000
> inode_dir_sub_section offset 1010 length 500
> inode_dir_sub_section offset 1010 length 500
> {code}
> Here you can see we still have the original section index, but then we also
> have sub-section entries that cover the entire section. Then a processor can
> either read the full section in serial, or read each sub-section in parallel.
> 2. In the Image Writer code, we should set a target number of sub-sections,
> and then based on the total inodes in memory, it will create that many
> sub-sections per major image section. I think the only sections worth doing
> this for are inode, inode_reference, inode_dir and snapshot_diff. All others
> tend to be fairly
[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index
[
https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877508#comment-16877508
]
He Xiaoqiao commented on HDFS-14617:
Just took a look, thanks a lot for this great work. It is very sensible. I
verify this patch, and mostly it works well,
1. check compatibility with old fsimage format,
* new version fsimage loader can load old format correctly, of course without
parallel loading since there are no INODE_SUB or INODE_DIR_SUB or other *_SUB
sections. the following fsimage saving will go with new format then parallel
loading if restart.
* old version fsimage loader can not load new format fsimage (as expected)
since it could not parse new section.
2. not benchmark with large fsimage file, will do it later.
some comments but all fairly minor implementation or style details.
1. Agree to disable parallel loading if config CompressionCodec for fsimage as
TODO comments;
2. Some metrics or more log for loading parallel section time cost will be
helpful to get this improvement.
3. Lambda expressions suggestions, for instance:
FSImageFormatPBINode#loadINodeDirectorySectionInParallel L226 could replace by
lambda,
{code:java}
for (FileSummary.Section s : sections) {
service.submit(() -> {
..
}
}
{code}
4. Need stricter inspect about INode numbers, INodeDirectorySection.DirEntry
number, etc when using parallel loading?
Thanks again for the great work.
> Improve fsimage load time by writing sub-sections to the fsimage index
> --
>
> Key: HDFS-14617
> URL: https://issues.apache.org/jira/browse/HDFS-14617
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Attachments: HDFS-14617.001.patch
>
>
> Loading an fsimage is basically a single threaded process. The current
> fsimage is written out in sections, eg iNode, iNode_Directory, Snapshots,
> Snapshot_Diff etc. Then at the end of the file, an index is written that
> contains the offset and length of each section. The image loader code uses
> this index to initialize an input stream to read and process each section. It
> is important that one section is fully loaded before another is started, as
> the next section depends on the results of the previous one.
> What I would like to propose is the following:
> 1. When writing the image, we can optionally output sub_sections to the
> index. That way, a given section would effectively be split into several
> sections, eg:
> {code:java}
>inode_section offset 10 length 1000
> inode_sub_section offset 10 length 500
> inode_sub_section offset 510 length 500
>
>inode_dir_section offset 1010 length 1000
> inode_dir_sub_section offset 1010 length 500
> inode_dir_sub_section offset 1010 length 500
> {code}
> Here you can see we still have the original section index, but then we also
> have sub-section entries that cover the entire section. Then a processor can
> either read the full section in serial, or read each sub-section in parallel.
> 2. In the Image Writer code, we should set a target number of sub-sections,
> and then based on the total inodes in memory, it will create that many
> sub-sections per major image section. I think the only sections worth doing
> this for are inode, inode_reference, inode_dir and snapshot_diff. All others
> tend to be fairly small in practice.
> 3. If there are under some threshold of inodes (eg 10M) then don't bother
> with the sub-sections as a serial load only takes a few seconds at that scale.
> 4. The image loading code can then have a switch to enable 'parallel loading'
> and a 'number of threads' where it uses the sub-sections, or if not enabled
> falls back to the existing logic to read the entire section in serial.
> Working with a large image of 316M inodes and 35GB on disk, I have a proof of
> concept of this change working, allowing just inode and inode_dir to be
> loaded in parallel, but I believe inode_reference and snapshot_diff can be
> make parallel with the same technique.
> Some benchmarks I have are as follows:
> {code:java}
> Threads 1 2 3 4
>
> inodes448 290 226 189
> inode_dir 326 211 170 161
> Total 927 651 535 488 (MD5 calculation about 100 seconds)
> {code}
> The above table shows the time in seconds to load the inode section and the
> inode_directory section, and then the total load time of the image.
> With 4 threads using the above technique, we are able to better than half the
> load time of the two sections. With the patch in HDFS-13694 it would take a
> further 100 seconds off the run time, going from 927 seconds to 388, which is
> a
[jira] [Commented] (HDFS-14628) Improve information on scanAndCompactStorages in BlockManager and lower log level
[ https://issues.apache.org/jira/browse/HDFS-14628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877504#comment-16877504 ] Shen Yinjie commented on HDFS-14628: [~ayushtkn] just found HDFS-13692 has fix this. > Improve information on scanAndCompactStorages in BlockManager and lower log > level > -- > > Key: HDFS-14628 > URL: https://issues.apache.org/jira/browse/HDFS-14628 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Shen Yinjie >Assignee: Shen Yinjie >Priority: Major > > BlockManager#scanAndCompactStorages is called every 600sec by default. > In big cluster with thousands of datanodes, it will print out 10 thousands of > informations every sec when scanAndCompactStorages() is running,which may > make much noise in namenode logs. And, currently these INFO logs at namenode > side and does not provide much information. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-14628) Improve information on scanAndCompactStorages in BlockManager and lower log level
[ https://issues.apache.org/jira/browse/HDFS-14628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shen Yinjie resolved HDFS-14628. Resolution: Duplicate Assignee: Shen Yinjie > Improve information on scanAndCompactStorages in BlockManager and lower log > level > -- > > Key: HDFS-14628 > URL: https://issues.apache.org/jira/browse/HDFS-14628 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Shen Yinjie >Assignee: Shen Yinjie >Priority: Major > > BlockManager#scanAndCompactStorages is called every 600sec by default. > In big cluster with thousands of datanodes, it will print out 10 thousands of > informations every sec when scanAndCompactStorages() is running,which may > make much noise in namenode logs. And, currently these INFO logs at namenode > side and does not provide much information. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1550) MiniOzoneCluster is not shutting down all the threads during shutdown.
[ https://issues.apache.org/jira/browse/HDDS-1550?focusedWorklogId=271395=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271395 ] ASF GitHub Bot logged work on HDDS-1550: Author: ASF GitHub Bot Created on: 03/Jul/19 05:30 Start Date: 03/Jul/19 05:30 Worklog Time Spent: 10m Work Description: mukul1987 commented on pull request #829: HDDS-1550. MiniOzoneCluster is not shutting down all the threads during shutdown. Contributed by Mukul Kumar Singh. URL: https://github.com/apache/hadoop/pull/829 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271395) Time Spent: 2h 20m (was: 2h 10m) > MiniOzoneCluster is not shutting down all the threads during shutdown. > -- > > Key: HDDS-1550 > URL: https://issues.apache.org/jira/browse/HDDS-1550 > Project: Hadoop Distributed Data Store > Issue Type: Test > Components: test >Affects Versions: 0.3.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > MiniOzoneCluster does not shutdown all the threads during shutdown. All the > threads must be shutdown to close the cluster correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1550) MiniOzoneCluster is not shutting down all the threads during shutdown.
[ https://issues.apache.org/jira/browse/HDDS-1550?focusedWorklogId=271394=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271394 ] ASF GitHub Bot logged work on HDDS-1550: Author: ASF GitHub Bot Created on: 03/Jul/19 05:29 Start Date: 03/Jul/19 05:29 Worklog Time Spent: 10m Work Description: mukul1987 commented on pull request #1050: HDDS-1550. MiniOzoneCluster is not shutting down all the threads during shutdown. Contributed by Mukul Kumar Singh. URL: https://github.com/apache/hadoop/pull/1050 Creating a new pull request. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271394) Time Spent: 2h 10m (was: 2h) > MiniOzoneCluster is not shutting down all the threads during shutdown. > -- > > Key: HDDS-1550 > URL: https://issues.apache.org/jira/browse/HDDS-1550 > Project: Hadoop Distributed Data Store > Issue Type: Test > Components: test >Affects Versions: 0.3.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > MiniOzoneCluster does not shutdown all the threads during shutdown. All the > threads must be shutdown to close the cluster correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1759) TestWatchForCommit crashes
Nanda kumar created HDDS-1759:
-
Summary: TestWatchForCommit crashes
Key: HDDS-1759
URL: https://issues.apache.org/jira/browse/HDDS-1759
Project: Hadoop Distributed Data Store
Issue Type: Test
Components: test
Reporter: Nanda kumar
{{org.apache.hadoop.ozone.client.rpc.TestWatchForCommit}} is crashing with the
following exception trace.
{noformat}
[ERROR] Crashed tests:
[ERROR] org.apache.hadoop.ozone.client.rpc.TestWatchForCommit
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException:
ExecutionException The forked VM terminated without properly saying goodbye. VM
crash or System.exit called?
[ERROR] Command was /bin/sh -c cd
/Users/nvadivelu/codebase/apache/hadoop/hadoop-ozone/integration-test &&
/Library/Java/JavaVirtualMachines/jdk1.8.0_152.jdk/Contents/Home/jre/bin/java
-Xmx2048m -XX:+HeapDumpOnOutOfMemoryError -jar
/Users/nvadivelu/codebase/apache/hadoop/hadoop-ozone/integration-test/target/surefire/surefirebooter6824244130326461346.jar
/Users/nvadivelu/codebase/apache/hadoop/hadoop-ozone/integration-test/target/surefire
2019-07-03T10-47-23_862-jvmRun1 surefire1503013258446082728tmp
surefire_07547129263746053478tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 1
[ERROR] Crashed tests:
[ERROR] org.apache.hadoop.ozone.client.rpc.TestWatchForCommit
[ERROR] at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:511)
[ERROR] at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:458)
[ERROR] at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:299)
[ERROR] at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:247)
[ERROR] at
org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1149)
[ERROR] at
org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:991)
[ERROR] at
org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:837)
[ERROR] at
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
[ERROR] at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
[ERROR] at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
[ERROR] at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
[ERROR] at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
[ERROR] at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
[ERROR] at
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
[ERROR] at
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
[ERROR] at
org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:309)
[ERROR] at
org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:194)
[ERROR] at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:107)
[ERROR] at org.apache.maven.cli.MavenCli.execute(MavenCli.java:955)
[ERROR] at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:290)
[ERROR] at org.apache.maven.cli.MavenCli.main(MavenCli.java:194)
[ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR] at java.lang.reflect.Method.invoke(Method.java:498)
[ERROR] at
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
[ERROR] at
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
[ERROR] at
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
[ERROR] at
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
[ERROR] Caused by:
org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM
terminated without properly saying goodbye. VM crash or System.exit called?
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1718) Increase Ratis Leader election timeout default to 10 seconds.
[ https://issues.apache.org/jira/browse/HDDS-1718?focusedWorklogId=271392=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271392 ] ASF GitHub Bot logged work on HDDS-1718: Author: ASF GitHub Bot Created on: 03/Jul/19 05:19 Start Date: 03/Jul/19 05:19 Worklog Time Spent: 10m Work Description: nandakumar131 commented on issue #1004: HDDS-1718 : Increase Ratis Leader election timeout default to 10 seconds URL: https://github.com/apache/hadoop/pull/1004#issuecomment-507942072 @avijayanhwx `TestCloseContainerCommandHandler` failure seems related to this change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271392) Time Spent: 1h (was: 50m) > Increase Ratis Leader election timeout default to 10 seconds. > - > > Key: HDDS-1718 > URL: https://issues.apache.org/jira/browse/HDDS-1718 > Project: Hadoop Distributed Data Store > Issue Type: Task > Components: Ozone Datanode >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Critical > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > While testing out ozone with long running clients which continuously write > data, it was noted that whenever a 1 second GC pause occurs in the leader, it > triggers a leader election thereby disturbing the steady state of the system > for more time than the GC pause delay. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14628) Improve information on scanAndCompactStorages in BlockManager and lower log level
[ https://issues.apache.org/jira/browse/HDFS-14628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877488#comment-16877488 ] Ayush Saxena commented on HDFS-14628: - Feels like can be reduced to DEBUG > Improve information on scanAndCompactStorages in BlockManager and lower log > level > -- > > Key: HDFS-14628 > URL: https://issues.apache.org/jira/browse/HDFS-14628 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Shen Yinjie >Priority: Major > > BlockManager#scanAndCompactStorages is called every 600sec by default. > In big cluster with thousands of datanodes, it will print out 10 thousands of > informations every sec when scanAndCompactStorages() is running,which may > make much noise in namenode logs. And, currently these INFO logs at namenode > side and does not provide much information. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14257) NPE when given the Invalid path to create target dir
[
https://issues.apache.org/jira/browse/HDFS-14257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877485#comment-16877485
]
hemanthboyina commented on HDFS-14257:
--
[~ramkumar] are you working on this issue ? if not can i takeover
> NPE when given the Invalid path to create target dir
>
>
> Key: HDFS-14257
> URL: https://issues.apache.org/jira/browse/HDFS-14257
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Harshakiran Reddy
>Assignee: venkata ramkumar
>Priority: Major
> Labels: RBF
>
> bin> ./hdfs dfs -mkdir hdfs://{color:red}hacluster2 /hacluster1{color}dest2/
> {noformat}
> -mkdir: Fatal internal error
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.FileSystem.fixRelativePart(FileSystem.java:2714)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.fixRelativePart(DistributedFileSystem.java:3229)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1618)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1742)
> at
> org.apache.hadoop.fs.shell.Mkdir.processNonexistentPath(Mkdir.java:74)
> at
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:287)
> at
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:269)
> at
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:121)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:391)
> bin>
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14628) Improve information on scanAndCompactStorages in BlockManager and lower log level
[ https://issues.apache.org/jira/browse/HDFS-14628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shen Yinjie updated HDFS-14628: --- Affects Version/s: 3.1.0 > Improve information on scanAndCompactStorages in BlockManager and lower log > level > -- > > Key: HDFS-14628 > URL: https://issues.apache.org/jira/browse/HDFS-14628 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Shen Yinjie >Priority: Major > > BlockManager#scanAndCompactStorages is called every 600sec by default. > In big cluster with thousands of datanodes, it will print out 10 thousands of > informations every sec when scanAndCompactStorages() is running,which may > make much noise in namenode logs. And, currently these INFO logs at namenode > side and does not provide much information. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14628) Improve information on scanAndCompactStorages in BlockManager and lower log level
Shen Yinjie created HDFS-14628: -- Summary: Improve information on scanAndCompactStorages in BlockManager and lower log level Key: HDFS-14628 URL: https://issues.apache.org/jira/browse/HDFS-14628 Project: Hadoop HDFS Issue Type: Improvement Reporter: Shen Yinjie BlockManager#scanAndCompactStorages is called every 600sec by default. In big cluster with thousands of datanodes, it will print out 10 thousands of informations every sec when scanAndCompactStorages() is running,which may make much noise in namenode logs. And, currently these INFO logs at namenode side and does not provide much information. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1303) Support native ACL for Ozone
[ https://issues.apache.org/jira/browse/HDDS-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-1303: Priority: Blocker (was: Major) > Support native ACL for Ozone > > > Key: HDDS-1303 > URL: https://issues.apache.org/jira/browse/HDDS-1303 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Ajay Kumar >Assignee: Anu Engineer >Priority: Blocker > > add native acl support for OM operations -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1758) Add replication and key deletion tests to MiniOzoneChaosCluster
[ https://issues.apache.org/jira/browse/HDDS-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDDS-1758: Status: Patch Available (was: Open) > Add replication and key deletion tests to MiniOzoneChaosCluster > --- > > Key: HDDS-1758 > URL: https://issues.apache.org/jira/browse/HDDS-1758 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This jira adds capability for deleting keys and also to test Replication > Manager code in MiniOzoneChaosCluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1758) Add replication and key deletion tests to MiniOzoneChaosCluster
[ https://issues.apache.org/jira/browse/HDDS-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-1758: - Labels: pull-request-available (was: ) > Add replication and key deletion tests to MiniOzoneChaosCluster > --- > > Key: HDDS-1758 > URL: https://issues.apache.org/jira/browse/HDDS-1758 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Labels: pull-request-available > > This jira adds capability for deleting keys and also to test Replication > Manager code in MiniOzoneChaosCluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1758) Add replication and key deletion tests to MiniOzoneChaosCluster
[ https://issues.apache.org/jira/browse/HDDS-1758?focusedWorklogId=271373=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271373 ] ASF GitHub Bot logged work on HDDS-1758: Author: ASF GitHub Bot Created on: 03/Jul/19 04:18 Start Date: 03/Jul/19 04:18 Worklog Time Spent: 10m Work Description: mukul1987 commented on pull request #1049: HDDS-1758. Add replication and key deletion tests to MiniOzoneChaosCluster. Contributed by Mukul Kumar Singh. URL: https://github.com/apache/hadoop/pull/1049 This patch adds key deletion and replication manager capability to MiniOzoneChaosCluster. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271373) Time Spent: 10m Remaining Estimate: 0h > Add replication and key deletion tests to MiniOzoneChaosCluster > --- > > Key: HDDS-1758 > URL: https://issues.apache.org/jira/browse/HDDS-1758 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This jira adds capability for deleting keys and also to test Replication > Manager code in MiniOzoneChaosCluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1673) smoketests are failing because an acl error
[
https://issues.apache.org/jira/browse/HDDS-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-1673:
Summary: smoketests are failing because an acl error (was: smoketests are
failig because an acl error)
> smoketests are failing because an acl error
> ---
>
> Key: HDDS-1673
> URL: https://issues.apache.org/jira/browse/HDDS-1673
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Ajay Kumar
>Priority: Blocker
>
> After executing this command:
> {code}
> yarn jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar pi 3 3 2
> {code}
> The result is:
> {code}
> Number of Maps = 3
> Samples per Map = 3
> 2019-06-12 03:16:20 ERROR OzoneClientFactory:294 - Couldn't create protocol
> class org.apache.hadoop.ozone.client.rpc.RpcClient exception:
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:291)
> at
> org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:169)
> at
> org.apache.hadoop.fs.ozone.BasicOzoneClientAdapterImpl.(BasicOzoneClientAdapterImpl.java:134)
> at
> org.apache.hadoop.fs.ozone.OzoneClientAdapterImpl.(OzoneClientAdapterImpl.java:50)
> at
> org.apache.hadoop.fs.ozone.OzoneFileSystem.createAdapter(OzoneFileSystem.java:103)
> at
> org.apache.hadoop.fs.ozone.BasicOzoneFileSystem.initialize(BasicOzoneFileSystem.java:143)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:463)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:522)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:275)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:360)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:368)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
> Caused by: org.apache.hadoop.hdds.conf.ConfigurationException: Can't inject
> configuration to class
> org.apache.hadoop.ozone.security.acl.OzoneAclConfig.setUserDefaultRights
> at
> org.apache.hadoop.hdds.conf.OzoneConfiguration.getObject(OzoneConfiguration.java:160)
> at
> org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:148)
> ... 36 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
>
[jira] [Updated] (HDDS-1718) Increase Ratis Leader election timeout default to 10 seconds.
[ https://issues.apache.org/jira/browse/HDDS-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-1718: Priority: Critical (was: Major) > Increase Ratis Leader election timeout default to 10 seconds. > - > > Key: HDDS-1718 > URL: https://issues.apache.org/jira/browse/HDDS-1718 > Project: Hadoop Distributed Data Store > Issue Type: Task > Components: Ozone Datanode >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Critical > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > While testing out ozone with long running clients which continuously write > data, it was noted that whenever a 1 second GC pause occurs in the leader, it > triggers a leader election thereby disturbing the steady state of the system > for more time than the GC pause delay. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1758) Add replication and key deletion tests to MiniOzoneChaosCluster
Mukul Kumar Singh created HDDS-1758: --- Summary: Add replication and key deletion tests to MiniOzoneChaosCluster Key: HDDS-1758 URL: https://issues.apache.org/jira/browse/HDDS-1758 Project: Hadoop Distributed Data Store Issue Type: Bug Components: test Affects Versions: 0.4.0 Reporter: Mukul Kumar Singh Assignee: Mukul Kumar Singh This jira adds capability for deleting keys and also to test Replication Manager code in MiniOzoneChaosCluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-200) Create Dead Node Watcher
[ https://issues.apache.org/jira/browse/HDDS-200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-200: --- Target Version/s: 0.5.0 (was: 0.4.1) > Create Dead Node Watcher > > > Key: HDDS-200 > URL: https://issues.apache.org/jira/browse/HDDS-200 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Ajay Kumar >Priority: Major > > This will be based on HDDS-195. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14627) Improvements to make slow archive storage works on HDFS
[ https://issues.apache.org/jira/browse/HDFS-14627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-14627: Attachment: (was: data_flow_between_datanode_and_aws_s3.jpg) > Improvements to make slow archive storage works on HDFS > --- > > Key: HDFS-14627 > URL: https://issues.apache.org/jira/browse/HDFS-14627 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Yang Yun >Priority: Minor > Attachments: data_flow_between_datanode_and_aws_s3.jpg > > > In our setup, we mount archival storage from remote. the write speed is about > 20M/Sec, the read speed is about 40M/Sec, and the normal file operations, for > example 'ls', are time consuming. > we add some improvements to make this kind of archive storage works in > currrent hdfs system. > 1. Add multiply to read/write timeout if block saved on archive storage. > 2. Save replica cache file of archive storage to other fast disk for quick > restart datanode, shutdownHook may does not execute if the saving takes too > long time. > 3. Check mount file system before using mounted archive storage. > 4. Reduce or avoid call DF during generating heartbeat report for archive > storage. > 5. Add option to skip archive block during decommission. > 6. Use multi-threads to scan archive storage. > 7. Check archive storage error with retry times. > 8. Add option to disable scan block on archive storage. > 9. Sleep a heartBeat time if there are too many difference when call > checkAndUpdate in DirectoryScanner > 10. An auto-service to scan fsimage and set the storage policy of files > according to policy. > 11. An auto-service to call mover to move the blocks to right storage. > 12. Dedup files on remote storage if the storage is reliable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14627) Improvements to make slow archive storage works on HDFS
[ https://issues.apache.org/jira/browse/HDFS-14627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-14627: Environment: (was: !data_flow_between_datanode_and_aws_s3.jpg!) > Improvements to make slow archive storage works on HDFS > --- > > Key: HDFS-14627 > URL: https://issues.apache.org/jira/browse/HDFS-14627 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Yang Yun >Priority: Minor > Attachments: data_flow_between_datanode_and_aws_s3.jpg > > > In our setup, we mount archival storage from remote. the write speed is about > 20M/Sec, the read speed is about 40M/Sec, and the normal file operations, for > example 'ls', are time consuming. > we add some improvements to make this kind of archive storage works in > currrent hdfs system. > 1. Add multiply to read/write timeout if block saved on archive storage. > 2. Save replica cache file of archive storage to other fast disk for quick > restart datanode, shutdownHook may does not execute if the saving takes too > long time. > 3. Check mount file system before using mounted archive storage. > 4. Reduce or avoid call DF during generating heartbeat report for archive > storage. > 5. Add option to skip archive block during decommission. > 6. Use multi-threads to scan archive storage. > 7. Check archive storage error with retry times. > 8. Add option to disable scan block on archive storage. > 9. Sleep a heartBeat time if there are too many difference when call > checkAndUpdate in DirectoryScanner > 10. An auto-service to scan fsimage and set the storage policy of files > according to policy. > 11. An auto-service to call mover to move the blocks to right storage. > 12. Dedup files on remote storage if the storage is reliable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14627) Improvements to make slow archive storage works on HDFS
[ https://issues.apache.org/jira/browse/HDFS-14627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-14627: Attachment: data_flow_between_datanode_and_aws_s3.jpg > Improvements to make slow archive storage works on HDFS > --- > > Key: HDFS-14627 > URL: https://issues.apache.org/jira/browse/HDFS-14627 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Yang Yun >Priority: Minor > Attachments: data_flow_between_datanode_and_aws_s3.jpg > > > In our setup, we mount archival storage from remote. the write speed is about > 20M/Sec, the read speed is about 40M/Sec, and the normal file operations, for > example 'ls', are time consuming. > we add some improvements to make this kind of archive storage works in > currrent hdfs system. > 1. Add multiply to read/write timeout if block saved on archive storage. > 2. Save replica cache file of archive storage to other fast disk for quick > restart datanode, shutdownHook may does not execute if the saving takes too > long time. > 3. Check mount file system before using mounted archive storage. > 4. Reduce or avoid call DF during generating heartbeat report for archive > storage. > 5. Add option to skip archive block during decommission. > 6. Use multi-threads to scan archive storage. > 7. Check archive storage error with retry times. > 8. Add option to disable scan block on archive storage. > 9. Sleep a heartBeat time if there are too many difference when call > checkAndUpdate in DirectoryScanner > 10. An auto-service to scan fsimage and set the storage policy of files > according to policy. > 11. An auto-service to call mover to move the blocks to right storage. > 12. Dedup files on remote storage if the storage is reliable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14627) Improvements to make slow archive storage works on HDFS
Yang Yun created HDFS-14627: --- Summary: Improvements to make slow archive storage works on HDFS Key: HDFS-14627 URL: https://issues.apache.org/jira/browse/HDFS-14627 Project: Hadoop HDFS Issue Type: Improvement Environment: !data_flow_between_datanode_and_aws_s3.jpg! Reporter: Yang Yun Attachments: data_flow_between_datanode_and_aws_s3.jpg In our setup, we mount archival storage from remote. the write speed is about 20M/Sec, the read speed is about 40M/Sec, and the normal file operations, for example 'ls', are time consuming. we add some improvements to make this kind of archive storage works in currrent hdfs system. 1. Add multiply to read/write timeout if block saved on archive storage. 2. Save replica cache file of archive storage to other fast disk for quick restart datanode, shutdownHook may does not execute if the saving takes too long time. 3. Check mount file system before using mounted archive storage. 4. Reduce or avoid call DF during generating heartbeat report for archive storage. 5. Add option to skip archive block during decommission. 6. Use multi-threads to scan archive storage. 7. Check archive storage error with retry times. 8. Add option to disable scan block on archive storage. 9. Sleep a heartBeat time if there are too many difference when call checkAndUpdate in DirectoryScanner 10. An auto-service to scan fsimage and set the storage policy of files according to policy. 11. An auto-service to call mover to move the blocks to right storage. 12. Dedup files on remote storage if the storage is reliable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14455) Fix typo in HAState.java
[ https://issues.apache.org/jira/browse/HDFS-14455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877460#comment-16877460 ] hunshenshi commented on HDFS-14455: --- [~nikhil.navadiya] thanks > Fix typo in HAState.java > > > Key: HDFS-14455 > URL: https://issues.apache.org/jira/browse/HDFS-14455 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Reporter: hunshenshi >Priority: Major > > There are some typo in HAState > destructuve -> destructive > Aleady -> Already > Transtion -> Transition -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1731) Implement File CreateFile Request to use Cache and DoubleBuffer
[
https://issues.apache.org/jira/browse/HDDS-1731?focusedWorklogId=271354=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271354
]
ASF GitHub Bot logged work on HDDS-1731:
Author: ASF GitHub Bot
Created on: 03/Jul/19 03:29
Start Date: 03/Jul/19 03:29
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #1044:
HDDS-1731. Implement File CreateFile Request to use Cache and DoubleBuffer.
URL: https://github.com/apache/hadoop/pull/1044#discussion_r299764094
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileCreateRequest.java
##
@@ -0,0 +1,348 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.request.file;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+
+import com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.FileEncryptionInfo;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.container.common.helpers.ExcludeList;
+import org.apache.hadoop.ozone.audit.OMAction;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OMMetrics;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest;
+import org.apache.hadoop.ozone.om.request.key.OMKeyRequest;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.CreateFileRequest;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.KeyArgs;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.OMRequest;
+import org.apache.hadoop.ozone.security.acl.IAccessAuthorizer;
+import org.apache.hadoop.ozone.security.acl.OzoneObj;
+import org.apache.hadoop.util.Time;
+import org.apache.hadoop.utils.UniqueId;
+import org.apache.hadoop.utils.db.Table;
+import org.apache.hadoop.utils.db.TableIterator;
+import org.apache.hadoop.utils.db.cache.CacheKey;
+import org.apache.hadoop.utils.db.cache.CacheValue;
+
+
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.DIRECTORY_EXISTS;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.DIRECTORY_EXISTS_IN_GIVENPATH;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.FILE_EXISTS_IN_GIVENPATH;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.FILE_EXISTS;
+import static
org.apache.hadoop.ozone.om.lock.OzoneManagerLock.Resource.BUCKET_LOCK;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.NONE;
+
+/**
+ * Handles create file request.
+ */
+public class OMFileCreateRequest extends OMKeyCreateRequest
+implements OMKeyRequest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OMFileCreateRequest.class);
+ public OMFileCreateRequest(OMRequest omRequest) {
+super(omRequest);
+ }
+
+
+ @Override
+ public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
+CreateFileRequest createFileRequest =
getOmRequest().getCreateFileRequest();
+Preconditions.checkNotNull(createFileRequest);
+
+KeyArgs keyArgs = createFileRequest.getKeyArgs();
+
+if (keyArgs.getKeyName().length() == 0) {
+ // Check if this is the root of the filesystem.
+ // Not throwing exception here, as need to throw exception after
+ // checking volume/bucket exists.
+ return
[jira] [Work logged] (HDDS-1731) Implement File CreateFile Request to use Cache and DoubleBuffer
[
https://issues.apache.org/jira/browse/HDDS-1731?focusedWorklogId=271353=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271353
]
ASF GitHub Bot logged work on HDDS-1731:
Author: ASF GitHub Bot
Created on: 03/Jul/19 03:25
Start Date: 03/Jul/19 03:25
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #1044:
HDDS-1731. Implement File CreateFile Request to use Cache and DoubleBuffer.
URL: https://github.com/apache/hadoop/pull/1044#discussion_r299763429
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileCreateRequest.java
##
@@ -0,0 +1,348 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.request.file;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+
+import com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.FileEncryptionInfo;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.container.common.helpers.ExcludeList;
+import org.apache.hadoop.ozone.audit.OMAction;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OMMetrics;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest;
+import org.apache.hadoop.ozone.om.request.key.OMKeyRequest;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.CreateFileRequest;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.KeyArgs;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.OMRequest;
+import org.apache.hadoop.ozone.security.acl.IAccessAuthorizer;
+import org.apache.hadoop.ozone.security.acl.OzoneObj;
+import org.apache.hadoop.util.Time;
+import org.apache.hadoop.utils.UniqueId;
+import org.apache.hadoop.utils.db.Table;
+import org.apache.hadoop.utils.db.TableIterator;
+import org.apache.hadoop.utils.db.cache.CacheKey;
+import org.apache.hadoop.utils.db.cache.CacheValue;
+
+
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.DIRECTORY_EXISTS;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.DIRECTORY_EXISTS_IN_GIVENPATH;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.FILE_EXISTS_IN_GIVENPATH;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.FILE_EXISTS;
+import static
org.apache.hadoop.ozone.om.lock.OzoneManagerLock.Resource.BUCKET_LOCK;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.NONE;
+
+/**
+ * Handles create file request.
+ */
+public class OMFileCreateRequest extends OMKeyCreateRequest
+implements OMKeyRequest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OMFileCreateRequest.class);
+ public OMFileCreateRequest(OMRequest omRequest) {
+super(omRequest);
+ }
+
+
+ @Override
+ public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
+CreateFileRequest createFileRequest =
getOmRequest().getCreateFileRequest();
+Preconditions.checkNotNull(createFileRequest);
+
+KeyArgs keyArgs = createFileRequest.getKeyArgs();
+
+if (keyArgs.getKeyName().length() == 0) {
+ // Check if this is the root of the filesystem.
+ // Not throwing exception here, as need to throw exception after
+ // checking volume/bucket exists.
+ return
[jira] [Work logged] (HDDS-1731) Implement File CreateFile Request to use Cache and DoubleBuffer
[
https://issues.apache.org/jira/browse/HDDS-1731?focusedWorklogId=271348=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271348
]
ASF GitHub Bot logged work on HDDS-1731:
Author: ASF GitHub Bot
Created on: 03/Jul/19 03:22
Start Date: 03/Jul/19 03:22
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #1044: HDDS-1731.
Implement File CreateFile Request to use Cache and DoubleBuffer.
URL: https://github.com/apache/hadoop/pull/1044#discussion_r299762796
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileCreateRequest.java
##
@@ -0,0 +1,348 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.request.file;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+
+import com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.FileEncryptionInfo;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.container.common.helpers.ExcludeList;
+import org.apache.hadoop.ozone.audit.OMAction;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OMMetrics;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest;
+import org.apache.hadoop.ozone.om.request.key.OMKeyRequest;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.CreateFileRequest;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.KeyArgs;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.OMRequest;
+import org.apache.hadoop.ozone.security.acl.IAccessAuthorizer;
+import org.apache.hadoop.ozone.security.acl.OzoneObj;
+import org.apache.hadoop.util.Time;
+import org.apache.hadoop.utils.UniqueId;
+import org.apache.hadoop.utils.db.Table;
+import org.apache.hadoop.utils.db.TableIterator;
+import org.apache.hadoop.utils.db.cache.CacheKey;
+import org.apache.hadoop.utils.db.cache.CacheValue;
+
+
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.DIRECTORY_EXISTS;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.DIRECTORY_EXISTS_IN_GIVENPATH;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.FILE_EXISTS_IN_GIVENPATH;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.FILE_EXISTS;
+import static
org.apache.hadoop.ozone.om.lock.OzoneManagerLock.Resource.BUCKET_LOCK;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.NONE;
+
+/**
+ * Handles create file request.
+ */
+public class OMFileCreateRequest extends OMKeyCreateRequest
+implements OMKeyRequest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OMFileCreateRequest.class);
+ public OMFileCreateRequest(OMRequest omRequest) {
+super(omRequest);
+ }
+
+
+ @Override
+ public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
+CreateFileRequest createFileRequest =
getOmRequest().getCreateFileRequest();
+Preconditions.checkNotNull(createFileRequest);
+
+KeyArgs keyArgs = createFileRequest.getKeyArgs();
+
+if (keyArgs.getKeyName().length() == 0) {
+ // Check if this is the root of the filesystem.
+ // Not throwing exception here, as need to throw exception after
+ // checking volume/bucket exists.
+ return
[jira] [Work logged] (HDDS-1731) Implement File CreateFile Request to use Cache and DoubleBuffer
[
https://issues.apache.org/jira/browse/HDDS-1731?focusedWorklogId=271347=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271347
]
ASF GitHub Bot logged work on HDDS-1731:
Author: ASF GitHub Bot
Created on: 03/Jul/19 03:20
Start Date: 03/Jul/19 03:20
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #1044:
HDDS-1731. Implement File CreateFile Request to use Cache and DoubleBuffer.
URL: https://github.com/apache/hadoop/pull/1044#discussion_r299762555
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileCreateRequest.java
##
@@ -0,0 +1,348 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.request.file;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+
+import com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.FileEncryptionInfo;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.container.common.helpers.ExcludeList;
+import org.apache.hadoop.ozone.audit.OMAction;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OMMetrics;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest;
+import org.apache.hadoop.ozone.om.request.key.OMKeyRequest;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.CreateFileRequest;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.KeyArgs;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.OMRequest;
+import org.apache.hadoop.ozone.security.acl.IAccessAuthorizer;
+import org.apache.hadoop.ozone.security.acl.OzoneObj;
+import org.apache.hadoop.util.Time;
+import org.apache.hadoop.utils.UniqueId;
+import org.apache.hadoop.utils.db.Table;
+import org.apache.hadoop.utils.db.TableIterator;
+import org.apache.hadoop.utils.db.cache.CacheKey;
+import org.apache.hadoop.utils.db.cache.CacheValue;
+
+
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.DIRECTORY_EXISTS;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.DIRECTORY_EXISTS_IN_GIVENPATH;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.FILE_EXISTS_IN_GIVENPATH;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.FILE_EXISTS;
+import static
org.apache.hadoop.ozone.om.lock.OzoneManagerLock.Resource.BUCKET_LOCK;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.NONE;
+
+/**
+ * Handles create file request.
+ */
+public class OMFileCreateRequest extends OMKeyCreateRequest
+implements OMKeyRequest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OMFileCreateRequest.class);
+ public OMFileCreateRequest(OMRequest omRequest) {
+super(omRequest);
+ }
+
+
+ @Override
+ public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
+CreateFileRequest createFileRequest =
getOmRequest().getCreateFileRequest();
+Preconditions.checkNotNull(createFileRequest);
+
+KeyArgs keyArgs = createFileRequest.getKeyArgs();
+
+if (keyArgs.getKeyName().length() == 0) {
+ // Check if this is the root of the filesystem.
+ // Not throwing exception here, as need to throw exception after
+ // checking volume/bucket exists.
+ return
[jira] [Work logged] (HDDS-1731) Implement File CreateFile Request to use Cache and DoubleBuffer
[
https://issues.apache.org/jira/browse/HDDS-1731?focusedWorklogId=271346=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271346
]
ASF GitHub Bot logged work on HDDS-1731:
Author: ASF GitHub Bot
Created on: 03/Jul/19 03:18
Start Date: 03/Jul/19 03:18
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #1044: HDDS-1731.
Implement File CreateFile Request to use Cache and DoubleBuffer.
URL: https://github.com/apache/hadoop/pull/1044#discussion_r299762339
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileCreateRequest.java
##
@@ -0,0 +1,348 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.request.file;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+
+import com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.FileEncryptionInfo;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.container.common.helpers.ExcludeList;
+import org.apache.hadoop.ozone.audit.OMAction;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OMMetrics;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest;
+import org.apache.hadoop.ozone.om.request.key.OMKeyRequest;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.CreateFileRequest;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.KeyArgs;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.OMRequest;
+import org.apache.hadoop.ozone.security.acl.IAccessAuthorizer;
+import org.apache.hadoop.ozone.security.acl.OzoneObj;
+import org.apache.hadoop.util.Time;
+import org.apache.hadoop.utils.UniqueId;
+import org.apache.hadoop.utils.db.Table;
+import org.apache.hadoop.utils.db.TableIterator;
+import org.apache.hadoop.utils.db.cache.CacheKey;
+import org.apache.hadoop.utils.db.cache.CacheValue;
+
+
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.DIRECTORY_EXISTS;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.DIRECTORY_EXISTS_IN_GIVENPATH;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.FILE_EXISTS_IN_GIVENPATH;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.FILE_EXISTS;
+import static
org.apache.hadoop.ozone.om.lock.OzoneManagerLock.Resource.BUCKET_LOCK;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.NONE;
+
+/**
+ * Handles create file request.
+ */
+public class OMFileCreateRequest extends OMKeyCreateRequest
+implements OMKeyRequest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OMFileCreateRequest.class);
+ public OMFileCreateRequest(OMRequest omRequest) {
+super(omRequest);
+ }
+
+
+ @Override
+ public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
+CreateFileRequest createFileRequest =
getOmRequest().getCreateFileRequest();
+Preconditions.checkNotNull(createFileRequest);
+
+KeyArgs keyArgs = createFileRequest.getKeyArgs();
+
+if (keyArgs.getKeyName().length() == 0) {
+ // Check if this is the root of the filesystem.
+ // Not throwing exception here, as need to throw exception after
+ // checking volume/bucket exists.
+ return
[jira] [Work logged] (HDDS-1731) Implement File CreateFile Request to use Cache and DoubleBuffer
[
https://issues.apache.org/jira/browse/HDDS-1731?focusedWorklogId=271344=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271344
]
ASF GitHub Bot logged work on HDDS-1731:
Author: ASF GitHub Bot
Created on: 03/Jul/19 03:18
Start Date: 03/Jul/19 03:18
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #1044: HDDS-1731.
Implement File CreateFile Request to use Cache and DoubleBuffer.
URL: https://github.com/apache/hadoop/pull/1044#discussion_r299762216
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileCreateRequest.java
##
@@ -0,0 +1,348 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.request.file;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+
+import com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.FileEncryptionInfo;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.container.common.helpers.ExcludeList;
+import org.apache.hadoop.ozone.audit.OMAction;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OMMetrics;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest;
+import org.apache.hadoop.ozone.om.request.key.OMKeyRequest;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.CreateFileRequest;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.KeyArgs;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.OMRequest;
+import org.apache.hadoop.ozone.security.acl.IAccessAuthorizer;
+import org.apache.hadoop.ozone.security.acl.OzoneObj;
+import org.apache.hadoop.util.Time;
+import org.apache.hadoop.utils.UniqueId;
+import org.apache.hadoop.utils.db.Table;
+import org.apache.hadoop.utils.db.TableIterator;
+import org.apache.hadoop.utils.db.cache.CacheKey;
+import org.apache.hadoop.utils.db.cache.CacheValue;
+
+
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.DIRECTORY_EXISTS;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.DIRECTORY_EXISTS_IN_GIVENPATH;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.FILE_EXISTS_IN_GIVENPATH;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.FILE_EXISTS;
+import static
org.apache.hadoop.ozone.om.lock.OzoneManagerLock.Resource.BUCKET_LOCK;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.NONE;
+
+/**
+ * Handles create file request.
+ */
+public class OMFileCreateRequest extends OMKeyCreateRequest
+implements OMKeyRequest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OMFileCreateRequest.class);
+ public OMFileCreateRequest(OMRequest omRequest) {
+super(omRequest);
+ }
+
+
+ @Override
+ public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
+CreateFileRequest createFileRequest =
getOmRequest().getCreateFileRequest();
+Preconditions.checkNotNull(createFileRequest);
+
+KeyArgs keyArgs = createFileRequest.getKeyArgs();
+
+if (keyArgs.getKeyName().length() == 0) {
+ // Check if this is the root of the filesystem.
+ // Not throwing exception here, as need to throw exception after
+ // checking volume/bucket exists.
+ return
[jira] [Work logged] (HDDS-1731) Implement File CreateFile Request to use Cache and DoubleBuffer
[
https://issues.apache.org/jira/browse/HDDS-1731?focusedWorklogId=271343=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271343
]
ASF GitHub Bot logged work on HDDS-1731:
Author: ASF GitHub Bot
Created on: 03/Jul/19 03:17
Start Date: 03/Jul/19 03:17
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #1044: HDDS-1731.
Implement File CreateFile Request to use Cache and DoubleBuffer.
URL: https://github.com/apache/hadoop/pull/1044#discussion_r299762127
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileCreateRequest.java
##
@@ -0,0 +1,348 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.request.file;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+
+import com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.FileEncryptionInfo;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.container.common.helpers.ExcludeList;
+import org.apache.hadoop.ozone.audit.OMAction;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OMMetrics;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest;
+import org.apache.hadoop.ozone.om.request.key.OMKeyRequest;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.CreateFileRequest;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.KeyArgs;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.OMRequest;
+import org.apache.hadoop.ozone.security.acl.IAccessAuthorizer;
+import org.apache.hadoop.ozone.security.acl.OzoneObj;
+import org.apache.hadoop.util.Time;
+import org.apache.hadoop.utils.UniqueId;
+import org.apache.hadoop.utils.db.Table;
+import org.apache.hadoop.utils.db.TableIterator;
+import org.apache.hadoop.utils.db.cache.CacheKey;
+import org.apache.hadoop.utils.db.cache.CacheValue;
+
+
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.DIRECTORY_EXISTS;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.DIRECTORY_EXISTS_IN_GIVENPATH;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.FILE_EXISTS_IN_GIVENPATH;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.FILE_EXISTS;
+import static
org.apache.hadoop.ozone.om.lock.OzoneManagerLock.Resource.BUCKET_LOCK;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.NONE;
+
+/**
+ * Handles create file request.
+ */
+public class OMFileCreateRequest extends OMKeyCreateRequest
+implements OMKeyRequest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OMFileCreateRequest.class);
+ public OMFileCreateRequest(OMRequest omRequest) {
+super(omRequest);
+ }
+
+
+ @Override
+ public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
+CreateFileRequest createFileRequest =
getOmRequest().getCreateFileRequest();
+Preconditions.checkNotNull(createFileRequest);
+
+KeyArgs keyArgs = createFileRequest.getKeyArgs();
+
+if (keyArgs.getKeyName().length() == 0) {
+ // Check if this is the root of the filesystem.
+ // Not throwing exception here, as need to throw exception after
+ // checking volume/bucket exists.
+ return
[jira] [Work logged] (HDDS-1384) TestBlockOutputStreamWithFailures is failing
[ https://issues.apache.org/jira/browse/HDDS-1384?focusedWorklogId=271339=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271339 ] ASF GitHub Bot logged work on HDDS-1384: Author: ASF GitHub Bot Created on: 03/Jul/19 02:49 Start Date: 03/Jul/19 02:49 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1029: HDDS-1384. TestBlockOutputStreamWithFailures is failing URL: https://github.com/apache/hadoop/pull/1029#issuecomment-507916770 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 109 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 69 | Maven dependency ordering for branch | | +1 | mvninstall | 543 | trunk passed | | +1 | compile | 265 | trunk passed | | +1 | checkstyle | 71 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 890 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 153 | trunk passed | | 0 | spotbugs | 307 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 494 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 29 | Maven dependency ordering for patch | | +1 | mvninstall | 429 | the patch passed | | +1 | compile | 249 | the patch passed | | +1 | javac | 249 | the patch passed | | -0 | checkstyle | 36 | hadoop-hdds: The patch generated 7 new + 0 unchanged - 0 fixed = 7 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 704 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 153 | the patch passed | | +1 | findbugs | 509 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 292 | hadoop-hdds in the patch passed. | | -1 | unit | 1560 | hadoop-ozone in the patch failed. | | +1 | asflicense | 42 | The patch does not generate ASF License warnings. | | | | 6804 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.client.rpc.TestBCSID | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.client.rpc.TestCommitWatcher | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=18.09.5 Server=18.09.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1029/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1029 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux fef3e86339d6 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 91cc197 | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1029/2/artifact/out/diff-checkstyle-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1029/2/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1029/2/testReport/ | | Max. process+thread count | 5407 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/container-service hadoop-ozone/integration-test U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1029/2/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271339) Time Spent: 2h 20m (was: 2h 10m) > TestBlockOutputStreamWithFailures is failing > > > Key: HDDS-1384 > URL: https://issues.apache.org/jira/browse/HDDS-1384 > Project: Hadoop Distributed Data Store >
[jira] [Commented] (HDFS-12914) Block report leases cause missing blocks until next report
[
https://issues.apache.org/jira/browse/HDFS-12914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877439#comment-16877439
]
star commented on HDFS-12914:
-
Yes, It is broken in branch-3.0 with 'private' indicator for 'processReport'.
There's no such issue for other banch HDFS-11673, in which 'private' indicator
is removed.
{quote} Collection processReport(
final DatanodeStorageInfo storageInfo,
final BlockListAsLongs report,
BlockReportContext context) throws IOException {{quote}
I am not sure whether branch-3.0 should be covered for this issue.
[~jojochuang], what do you think?
> Block report leases cause missing blocks until next report
> --
>
> Key: HDFS-12914
> URL: https://issues.apache.org/jira/browse/HDFS-12914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0, 2.9.2
>Reporter: Daryn Sharp
>Assignee: Santosh Marella
>Priority: Critical
> Fix For: 3.0.4, 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-12914-branch-2.001.patch,
> HDFS-12914-trunk.00.patch, HDFS-12914-trunk.01.patch, HDFS-12914.005.patch,
> HDFS-12914.006.patch, HDFS-12914.007.patch, HDFS-12914.008.patch,
> HDFS-12914.009.patch, HDFS-12914.branch-2.patch,
> HDFS-12914.branch-3.1.001.patch, HDFS-12914.branch-3.1.002.patch,
> HDFS-12914.branch-3.2.patch, HDFS-12914.utfix.patch
>
>
> {{BlockReportLeaseManager#checkLease}} will reject FBRs from DNs for
> conditions such as "unknown datanode", "not in pending set", "lease has
> expired", wrong lease id, etc. Lease rejection does not throw an exception.
> It returns false which bubbles up to {{NameNodeRpcServer#blockReport}} and
> interpreted as {{noStaleStorages}}.
> A re-registering node whose FBR is rejected from an invalid lease becomes
> active with _no blocks_. A replication storm ensues possibly causing DNs to
> temporarily go dead (HDFS-12645), leading to more FBR lease rejections on
> re-registration. The cluster will have many "missing blocks" until the DNs
> next FBR is sent and/or forced.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12914) Block report leases cause missing blocks until next report
[
https://issues.apache.org/jira/browse/HDFS-12914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877439#comment-16877439
]
star edited comment on HDFS-12914 at 7/3/19 2:43 AM:
-
Yes, It is broken in branch-3.0 with 'private' indicator for 'processReport'.
There's no such issue for other branch HDFS-11673, in which 'private' indicator
is removed.
{quote} Collection processReport(
final DatanodeStorageInfo storageInfo,
final BlockListAsLongs report,
BlockReportContext context) throws IOException {
{quote}
I am not sure whether branch-3.0 should be covered for this issue.
[~jojochuang], what do you think?
was (Author: starphin):
Yes, It is broken in branch-3.0 with 'private' indicator for 'processReport'.
There's no such issue for other banch HDFS-11673, in which 'private' indicator
is removed.
{quote} Collection processReport(
final DatanodeStorageInfo storageInfo,
final BlockListAsLongs report,
BlockReportContext context) throws IOException {{quote}
I am not sure whether branch-3.0 should be covered for this issue.
[~jojochuang], what do you think?
> Block report leases cause missing blocks until next report
> --
>
> Key: HDFS-12914
> URL: https://issues.apache.org/jira/browse/HDFS-12914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0, 2.9.2
>Reporter: Daryn Sharp
>Assignee: Santosh Marella
>Priority: Critical
> Fix For: 3.0.4, 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-12914-branch-2.001.patch,
> HDFS-12914-trunk.00.patch, HDFS-12914-trunk.01.patch, HDFS-12914.005.patch,
> HDFS-12914.006.patch, HDFS-12914.007.patch, HDFS-12914.008.patch,
> HDFS-12914.009.patch, HDFS-12914.branch-2.patch,
> HDFS-12914.branch-3.1.001.patch, HDFS-12914.branch-3.1.002.patch,
> HDFS-12914.branch-3.2.patch, HDFS-12914.utfix.patch
>
>
> {{BlockReportLeaseManager#checkLease}} will reject FBRs from DNs for
> conditions such as "unknown datanode", "not in pending set", "lease has
> expired", wrong lease id, etc. Lease rejection does not throw an exception.
> It returns false which bubbles up to {{NameNodeRpcServer#blockReport}} and
> interpreted as {{noStaleStorages}}.
> A re-registering node whose FBR is rejected from an invalid lease becomes
> active with _no blocks_. A replication storm ensues possibly causing DNs to
> temporarily go dead (HDFS-12645), leading to more FBR lease rejections on
> re-registration. The cluster will have many "missing blocks" until the DNs
> next FBR is sent and/or forced.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14290) Unexpected message type: PooledUnsafeDirectByteBuf when get datanode info by DatanodeWebHdfsMethods
[ https://issues.apache.org/jira/browse/HDFS-14290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877409#comment-16877409 ] Lisheng Sun commented on HDFS-14290: Sorry, I don't reproduce it when update new netty. The problem should cause by old version netty. > Unexpected message type: PooledUnsafeDirectByteBuf when get datanode info by > DatanodeWebHdfsMethods > --- > > Key: HDFS-14290 > URL: https://issues.apache.org/jira/browse/HDFS-14290 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, webhdfs >Affects Versions: 2.7.0, 2.7.1 >Reporter: Lisheng Sun >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14290.000.patch, webhdfs show.png > > > The issue is there is no HttpRequestDecoder in InboundHandler of netty, > appear unexpected message type when read message. > > !webhdfs show.png! > DEBUG org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Proxy > failed. Cause: > com.xiaomi.infra.thirdparty.io.netty.handler.codec.EncoderException: > java.lang.IllegalStateException: unexpected message type: > PooledUnsafeDirectByteBuf > at > com.xiaomi.infra.thirdparty.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) > at > com.xiaomi.infra.thirdparty.io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) > at > com.xiaomi.infra.thirdparty.io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:304) > at > com.xiaomi.infra.thirdparty.io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:137) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) > at > com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1051) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) > at > org.apache.hadoop.hdfs.server.datanode.web.SimpleHttpProxyHandler$Forwarder.channelRead(SimpleHttpProxyHandler.java:80) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) > at > com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1414) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) > at > com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:945) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:146) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) > at >
[jira] [Commented] (HDFS-14483) Backport HDFS-14111,HDFS-3246 ByteBuffer pread interface to branch-2.9
[ https://issues.apache.org/jira/browse/HDFS-14483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877403#comment-16877403 ] Lisheng Sun commented on HDFS-14483: Hi [~stack] test failures is unrelated to this patch. Thank you. > Backport HDFS-14111,HDFS-3246 ByteBuffer pread interface to branch-2.9 > -- > > Key: HDFS-14483 > URL: https://issues.apache.org/jira/browse/HDFS-14483 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Zheng Hu >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14483.branch-2.8.v1.patch, > HDFS-14483.branch-2.9.v1.patch, HDFS-14483.branch-2.9.v1.patch, > HDFS-14483.branch-2.9.v2.patch, HDFS-14483.branch-2.9.v2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14483) Backport HDFS-14111,HDFS-3246 ByteBuffer pread interface to branch-2.9
[ https://issues.apache.org/jira/browse/HDFS-14483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877402#comment-16877402 ] stack commented on HDFS-14483: -- Are the test failures related [~leosun08]? Thanks. > Backport HDFS-14111,HDFS-3246 ByteBuffer pread interface to branch-2.9 > -- > > Key: HDFS-14483 > URL: https://issues.apache.org/jira/browse/HDFS-14483 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Zheng Hu >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14483.branch-2.8.v1.patch, > HDFS-14483.branch-2.9.v1.patch, HDFS-14483.branch-2.9.v1.patch, > HDFS-14483.branch-2.9.v2.patch, HDFS-14483.branch-2.9.v2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14313) Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du
[ https://issues.apache.org/jira/browse/HDFS-14313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877400#comment-16877400 ] Lisheng Sun commented on HDFS-14313: ping [~jojochuang] Could you have time to review this patch? Thank you. > Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory > instead of df/du > > > Key: HDFS-14313 > URL: https://issues.apache.org/jira/browse/HDFS-14313 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, performance >Affects Versions: 2.6.0, 2.7.0, 2.8.0, 2.9.0, 3.0.0, 3.1.0 >Reporter: Lisheng Sun >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14313.000.patch, HDFS-14313.001.patch, > HDFS-14313.002.patch, HDFS-14313.003.patch, HDFS-14313.004.patch > > > There are two ways of DU/DF getting used space that are insufficient. > # Running DU across lots of disks is very expensive and running all of the > processes at the same time creates a noticeable IO spike. > # Running DF is inaccurate when the disk sharing by multiple datanode or > other servers. > Getting hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfos in memory > is very small and accurate. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14615) NPE writing edit logs
[
https://issues.apache.org/jira/browse/HDFS-14615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877398#comment-16877398
]
Lisheng Sun commented on HDFS-14615:
Hi [~jojochuang] I work on this problem and assign to me directly, hope you
don't mind,Thank you.
> NPE writing edit logs
> -
>
> Key: HDFS-14615
> URL: https://issues.apache.org/jira/browse/HDFS-14615
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.1.1
>Reporter: Wei-Chiu Chuang
>Assignee: Lisheng Sun
>Priority: Major
>
> Hit a weird bug where writing mkdir op to edit log throws an NPE and NameNode
> crashed
> {noformat}
> 2019-06-26 10:57:27,398 ERROR
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Error: write op failed for
> (journal
> JournalAndStream(mgr=FileJournalManager(root=/ssd/work/src/upstream/impala/testdata/cluster/cdh6/node-1/data/dfs/nn),
>
> stream=EditLogFileOutputStream(/ssd/work/src/upstream/impala/testdata/cluster/cdh6/node-1/data/dfs/nn/current/edits_inprogress_0598588)))
> java.lang.NullPointerException
> at org.apache.hadoop.io.Text.encode(Text.java:451)
> at org.apache.hadoop.io.Text.encode(Text.java:431)
> at org.apache.hadoop.io.Text.writeString(Text.java:491)
> at
> org.apache.hadoop.fs.permission.PermissionStatus.write(PermissionStatus.java:104)
> at
> org.apache.hadoop.fs.permission.PermissionStatus.write(PermissionStatus.java:84)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$MkdirOp.writeFields(FSEditLogOp.java:1654)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$Writer.writeOp(FSEditLogOp.java:4866)
> at
> org.apache.hadoop.hdfs.server.namenode.EditsDoubleBuffer$TxnBuffer.writeOp(EditsDoubleBuffer.java:157)
> at
> org.apache.hadoop.hdfs.server.namenode.EditsDoubleBuffer.writeOp(EditsDoubleBuffer.java:60)
> at
> org.apache.hadoop.hdfs.server.namenode.EditLogFileOutputStream.write(EditLogFileOutputStream.java:97)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream$1.apply(JournalSet.java:444)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:385)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.access$100(JournalSet.java:55)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream.write(JournalSet.java:440)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.doEditTransaction(FSEditLog.java:481)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync$Edit.logEdit(FSEditLogAsync.java:288)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.run(FSEditLogAsync.java:232)
> {noformat}
> The stacktrace is similar to SENTRY-555, which is thought to be a Sentry bug
> (authorization provider), but this cluster doesn't have Sentry and therefore
> could be a genuine HDFS bug.
> File this jira to keep a record.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14615) NPE writing edit logs
[
https://issues.apache.org/jira/browse/HDFS-14615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lisheng Sun reassigned HDFS-14615:
--
Assignee: Lisheng Sun
> NPE writing edit logs
> -
>
> Key: HDFS-14615
> URL: https://issues.apache.org/jira/browse/HDFS-14615
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.1.1
>Reporter: Wei-Chiu Chuang
>Assignee: Lisheng Sun
>Priority: Major
>
> Hit a weird bug where writing mkdir op to edit log throws an NPE and NameNode
> crashed
> {noformat}
> 2019-06-26 10:57:27,398 ERROR
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Error: write op failed for
> (journal
> JournalAndStream(mgr=FileJournalManager(root=/ssd/work/src/upstream/impala/testdata/cluster/cdh6/node-1/data/dfs/nn),
>
> stream=EditLogFileOutputStream(/ssd/work/src/upstream/impala/testdata/cluster/cdh6/node-1/data/dfs/nn/current/edits_inprogress_0598588)))
> java.lang.NullPointerException
> at org.apache.hadoop.io.Text.encode(Text.java:451)
> at org.apache.hadoop.io.Text.encode(Text.java:431)
> at org.apache.hadoop.io.Text.writeString(Text.java:491)
> at
> org.apache.hadoop.fs.permission.PermissionStatus.write(PermissionStatus.java:104)
> at
> org.apache.hadoop.fs.permission.PermissionStatus.write(PermissionStatus.java:84)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$MkdirOp.writeFields(FSEditLogOp.java:1654)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$Writer.writeOp(FSEditLogOp.java:4866)
> at
> org.apache.hadoop.hdfs.server.namenode.EditsDoubleBuffer$TxnBuffer.writeOp(EditsDoubleBuffer.java:157)
> at
> org.apache.hadoop.hdfs.server.namenode.EditsDoubleBuffer.writeOp(EditsDoubleBuffer.java:60)
> at
> org.apache.hadoop.hdfs.server.namenode.EditLogFileOutputStream.write(EditLogFileOutputStream.java:97)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream$1.apply(JournalSet.java:444)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:385)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.access$100(JournalSet.java:55)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream.write(JournalSet.java:440)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.doEditTransaction(FSEditLog.java:481)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync$Edit.logEdit(FSEditLogAsync.java:288)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.run(FSEditLogAsync.java:232)
> {noformat}
> The stacktrace is similar to SENTRY-555, which is thought to be a Sentry bug
> (authorization provider), but this cluster doesn't have Sentry and therefore
> could be a genuine HDFS bug.
> File this jira to keep a record.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14483) Backport HDFS-14111,HDFS-3246 ByteBuffer pread interface to branch-2.9
[
https://issues.apache.org/jira/browse/HDFS-14483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877390#comment-16877390
]
Lisheng Sun commented on HDFS-14483:
Thank [~stack] for your comments.
{quote}Comparing the BB-based decrypt to the byte[] version, this confuses me:
buf.limit(start + len + Math.min(n - len, localInBuffer.remaining()));
In the byte[] version, its len - n vs n - len figuring how much to decrypt. I
see the byte[] decrypt loop is n < length vs the bb decrypt which is len < n...
so I think its fine but take a look please.
{quote}
I think loop should be decryptedBytes < remain data to be decrypted between
length the byte[] decrypt and the bb decrypt, so I update the variable name
that is more clear semantics.
{quote}Should these resets be in the finally block just below?
425 buf.position(start + len);
426 buf.limit(limit);
{quote}
These code should be placed here ,because only decrpt successfully, call
buf.position(), buf.limit() and put the decrypted data into the buffer.
{quote} Nice javadoc in ByteBufferPositionedReadable
{quote}
Thank you for your suggestion.I add Javadoc in ByteBufferPositionedReadable.
{quote}nit: more permutations in testPositionedReadWithByteBuffer would be
nice-to-have – though the testByteBufferPread addition is good. Maybe a
follow-on? Could test more edge cases... a failed read or a read that does not
fill the read request amount? The positionedReadCheckWithByteBuffer is nice.
Nice addition of the strncmp check in the test_libhdfs_ops.c file and bulking
up of the pread tests.
Yeah, skip the re-formatting of unrelated code (especially when adds mistake as
in '916 method = "open";')... which adds an offset.
Nice comments added to c function names.
{quote}
I have recovered re-formatting unrelated code and add comments to c function
names.
I will pay attention to these problems and add more edge cases.
Please correct me if I am wrong. Could you continue to help review code? Thank
[~stack] again.
> Backport HDFS-14111,HDFS-3246 ByteBuffer pread interface to branch-2.9
> --
>
> Key: HDFS-14483
> URL: https://issues.apache.org/jira/browse/HDFS-14483
> Project: Hadoop HDFS
> Issue Type: Task
>Reporter: Zheng Hu
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-14483.branch-2.8.v1.patch,
> HDFS-14483.branch-2.9.v1.patch, HDFS-14483.branch-2.9.v1.patch,
> HDFS-14483.branch-2.9.v2.patch, HDFS-14483.branch-2.9.v2.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1735) Create separate unit and integration test executor dev-support script
[
https://issues.apache.org/jira/browse/HDDS-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877382#comment-16877382
]
Anu Engineer edited comment on HDDS-1735 at 7/3/19 12:59 AM:
-
{code:java}
./hadoop-common-project/hadoop-auth/dev-support/findbugsExcludeFile.xml
./hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml
./hadoop-common-project/hadoop-nfs/dev-support/findbugsExcludeFile.xml
./hadoop-common-project/hadoop-minikdc/dev-support/findbugsExcludeFile.xml
./hadoop-common-project/hadoop-kms/dev-support/findbugsExcludeFile.xml
./hadoop-hdfs-project/hadoop-hdfs-httpfs/dev-support/findbugsExcludeFile.xml
./hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml
./hadoop-hdfs-project/hadoop-hdfs-nfs/dev-support/findbugsExcludeFile.xml
./hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml
./hadoop-hdfs-project/hadoop-hdfs-rbf/dev-support/findbugsExcludeFile.xml{code}
what are these but suppressions ? Allow me to show you some more examples:
Here is some code from one these files:
{code:java}
{code}
This is code written by someone who made a judgement call, they saw the warning
message from findbugs and since they are experts on this domain made a
judgement call that they know better than a tool. Unless you have evidence to
prove that these lines are wrong; it would be better not to judge these
developers. They contribute work, time and talent to our cause; and you are
saying that grepping few lines over the code allows you to make a judgement
call and deem that this is inferior code? I have shown your claim that Hadoop
does not use suppressions – and the only one was contributed by [~elek] is
incorrect. The above lines prove that suppression is rampant in Hadoop code
base. Much more than Ozone, if you care to look.
What I don't understand is why have such blatant disregard and disrespect for
the community members? if you find a bug, file it as JIRA, please stop making
blanket statements and over-arching pronouncements on code quality. Let us be
specific; if you see an issue; we welcome it; and thank you for your making our
software better; the nature of software development is such that we will have
issues. But let us do it with some consideration; and not say things like I
grepped and found suppressions; therefore your code is bad. if you don't
understand that code; feel free to ask us; perhaps we are wrong; and you have
found a bug; perhaps we have a perfectly valid reason to do it that way.
Once more; I stand by my earlier statement; unless you are willing to measure
and show me numbers – I am going to presume that you have really no issues to
be solved.
Thank you for the conversation.
was (Author: anu):
{code:java}
./hadoop-common-project/hadoop-auth/dev-support/findbugsExcludeFile.xml
./hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml
./hadoop-common-project/hadoop-nfs/dev-support/findbugsExcludeFile.xml
./hadoop-common-project/hadoop-minikdc/dev-support/findbugsExcludeFile.xml
./hadoop-common-project/hadoop-kms/dev-support/findbugsExcludeFile.xml
./hadoop-hdds/common/target/findbugsExcludeFile.xml
./hadoop-hdds/common/dev-support/findbugsExcludeFile.xml
./hadoop-hdds/container-service/target/findbugsExcludeFile.xml
./hadoop-hdds/container-service/dev-support/findbugsExcludeFile.xml
./hadoop-hdfs-project/hadoop-hdfs-httpfs/dev-support/findbugsExcludeFile.xml
./hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml
./hadoop-hdfs-project/hadoop-hdfs-nfs/dev-support/findbugsExcludeFile.xml
./hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml
./hadoop-hdfs-project/hadoop-hdfs-rbf/dev-support/findbugsExcludeFile.xml{code}
what are these but suppressions ? Allow me to show you some more examples:
Here is some code from one these files:
{code:java}
{code}
This is code written by someone who made a judgement call, they saw the warning
message from findbugs and since they are experts on this domain made a
judgement call that they know better than a tool. Unless you have evidence to
prove that these lines are wrong; it would be better not to judge these
developers. They contribute work, time and talent to our cause; and you are
saying that grepping few lines over the code allows you to make a judgement
call and deem that this is inferior code? I have shown your claim that Hadoop
does not use suppressions – and the only one was contributed by [~elek] is
incorrect. The above lines prove that suppression is rampant in Hadoop code
base. Much more than Ozone, if you care to look.
What I don't understand is why have such blatant disregard and disrespect for
the community members? if you find a bug, file it as JIRA,
[jira] [Comment Edited] (HDDS-1661) Consolidate hadoop-hdds and hadoop-ozone into hadoop-ozone-project
[
https://issues.apache.org/jira/browse/HDDS-1661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876667#comment-16876667
]
Anu Engineer edited comment on HDDS-1661 at 7/3/19 12:55 AM:
-
I think that would be too early now. Either Ozone should be as good as HDFS or
HDFS should start using HDDS.These are the two possible out comes we are
looking for.
HDFS cannot start using HDDS since HDFS did not want to take dependency on
something that is not proven in the field.
IMHO, the best time to have a discussion about should we move HDFS over HDDS
or is Ozone good enough? Needs us to prove that HDDS is stable and performant.
So perhaps after the Ozone GA?
was (Author: anu):
I think that would be too early now. Either Ozone should be as good as HDFS or
HDFS should start using HDDS.These are the two possible out comes we are
looking for.
HDFS cannot start using HDDS since HDFS did not want to take dependency on
something that is not prove in the field.
IMHO, the best time to have a discussion about should we move HDFS over HDDS
or is Ozone good enough? Needs us to prove that HDDS is stable and performant.
So perhaps after the Ozone GA?
> Consolidate hadoop-hdds and hadoop-ozone into hadoop-ozone-project
> --
>
> Key: HDDS-1661
> URL: https://issues.apache.org/jira/browse/HDDS-1661
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Eric Yang
>Priority: Major
>
> Ozone source code is some what fragmented in Hadoop source code. The current
> code looks like:
> {code}
> hadoop/pom.ozone.xml
> ├── hadoop-hdds
> └── hadoop-ozone
> {code}
> It is helpful to consolidate the project into high level grouping such as:
> {code}
> hadoop
> └── hadoop-ozone-project/pom.xml
> └── hadoop-ozone-project/hadoop-hdds
> └── hadoop-ozone-project/hadoop-ozone
> {code}
> This allows user to build ozone from hadoop-ozone-project directory.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1735) Create separate unit and integration test executor dev-support script
[ https://issues.apache.org/jira/browse/HDDS-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877386#comment-16877386 ] Anu Engineer commented on HDDS-1735: and I plan to commit this patch, as soon as the shell check issues are addressed. [~elek] thanks > Create separate unit and integration test executor dev-support script > - > > Key: HDDS-1735 > URL: https://issues.apache.org/jira/browse/HDDS-1735 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Attachments: Screen Shot 2019-07-02 at 3.25.33 PM.png > > Time Spent: 1h 10m > Remaining Estimate: 0h > > hadoop-ozone/dev-support/checks directory contains multiple helper script to > execute different type of testing (findbugs, rat, unit, build). > They easily define how tests should be executed, with the following contract: > * The problems should be printed out to the console > * in case of test failure a non zero exit code should be used > > The tests are working well (in fact I have some experiments with executing > these scripts on k8s and argo where all the shell scripts are executed > parallel) but we need some update: > 1. Most important: the unit tests and integration tests can be separated. > Integration tests are more flaky and it's better to have a way to run only > the normal unit tests > 2. As HDDS-1115 introduced a pom.ozone.xml it's better to use them instead > of the magical "am pl hadoop-ozone-dist" trick-- > 3. To make it possible to run blockade test in containers we should use - T > flag with docker-compose > 4. checkstyle violations are printed out to the console -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1735) Create separate unit and integration test executor dev-support script
[
https://issues.apache.org/jira/browse/HDDS-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877382#comment-16877382
]
Anu Engineer edited comment on HDDS-1735 at 7/3/19 12:42 AM:
-
{code:java}
./hadoop-common-project/hadoop-auth/dev-support/findbugsExcludeFile.xml
./hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml
./hadoop-common-project/hadoop-nfs/dev-support/findbugsExcludeFile.xml
./hadoop-common-project/hadoop-minikdc/dev-support/findbugsExcludeFile.xml
./hadoop-common-project/hadoop-kms/dev-support/findbugsExcludeFile.xml
./hadoop-hdds/common/target/findbugsExcludeFile.xml
./hadoop-hdds/common/dev-support/findbugsExcludeFile.xml
./hadoop-hdds/container-service/target/findbugsExcludeFile.xml
./hadoop-hdds/container-service/dev-support/findbugsExcludeFile.xml
./hadoop-hdfs-project/hadoop-hdfs-httpfs/dev-support/findbugsExcludeFile.xml
./hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml
./hadoop-hdfs-project/hadoop-hdfs-nfs/dev-support/findbugsExcludeFile.xml
./hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml
./hadoop-hdfs-project/hadoop-hdfs-rbf/dev-support/findbugsExcludeFile.xml{code}
what are these but suppressions ? Allow me to show you some more examples:
Here is some code from one these files:
{code:java}
{code}
This is code written by someone who made a judgement call, they saw the warning
message from findbugs and since they are experts on this domain made a
judgement call that they know better than a tool. Unless you have evidence to
prove that these lines are wrong; it would be better not to judge these
developers. They contribute work, time and talent to our cause; and you are
saying that grepping few lines over the code allows you to make a judgement
call and deem that this is inferior code? I have shown your claim that Hadoop
does not use suppressions – and the only one was contributed by [~elek] is
incorrect. The above lines prove that suppression is rampant in Hadoop code
base. Much more than Ozone, if you care to look.
What I don't understand is why have such blatant disregard and disrespect for
the community members? if you find a bug, file it as JIRA, please stop making
blanket statements and over-arching pronouncements on code quality. Let us be
specific; if you see an issue; we welcome it; and thank you for your making our
software better; the nature of software development is such that we will have
issues. But let us do it with some consideration; and not say things like I
grepped and found suppressions; therefore your code is bad. if you don't
understand that code; feel free to ask us; perhaps we are wrong; and you have
found a bug; perhaps we have a perfectly valid reason to do it that way.
Once more; I stand by my earlier statement; unless you are willing to measure
and show me numbers – I am going to presume that you have really no issues to
be solved.
Thank you for the conversation.
was (Author: anu):
{code:java}
./hadoop-common-project/hadoop-auth/dev-support/findbugsExcludeFile.xml
./hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml
./hadoop-common-project/hadoop-nfs/dev-support/findbugsExcludeFile.xml
./hadoop-common-project/hadoop-minikdc/dev-support/findbugsExcludeFile.xml
./hadoop-common-project/hadoop-kms/dev-support/findbugsExcludeFile.xml
./hadoop-hdds/common/target/findbugsExcludeFile.xml
./hadoop-hdds/common/dev-support/findbugsExcludeFile.xml
./hadoop-hdds/container-service/target/findbugsExcludeFile.xml
./hadoop-hdds/container-service/dev-support/findbugsExcludeFile.xml
./hadoop-hdfs-project/hadoop-hdfs-httpfs/dev-support/findbugsExcludeFile.xml
./hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml
./hadoop-hdfs-project/hadoop-hdfs-nfs/dev-support/findbugsExcludeFile.xml
./hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml
./hadoop-hdfs-project/hadoop-hdfs-rbf/dev-support/findbugsExcludeFile.xml{code}
what are these but suppressions ? Allow me to show you some more examples:
Here is some code from one these files:
{code:java}
{code}
This is code written by someone who made a judgement call, they saw the warning
message from findbugs and since they are experts on this domain made a
judgement call that they know better than a tool. Unless you have evidence to
prove that these lines are wrong; it would be better not to judge these
developers. They contribute work, time and talent to our cause; and you are
saying that grepping few lines over the code allows you to make a judgement
call and deem that this is inferior code? I have shown your claim that Hadoop
does not use suppressions – and the only one was contributed by [~elek] is
incorrect. The above
[jira] [Commented] (HDDS-1554) Create disk tests for fault injection test
[
https://issues.apache.org/jira/browse/HDDS-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877385#comment-16877385
]
Hadoop QA commented on HDDS-1554:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
51s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} yamllint {color} | {color:blue} 0m
0s{color} | {color:blue} yamllint was not available. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
1s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 30 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 3s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
10s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 6m
24s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 10m
31s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
17s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 1s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 5m 47s{color}
| {color:red} hadoop-hdds in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 28m 54s{color}
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
7s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}127m 32s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdds.scm.block.TestBlockManager |
| | hadoop.ozone.TestMiniOzoneCluster |
| |
[jira] [Commented] (HDDS-1735) Create separate unit and integration test executor dev-support script
[
https://issues.apache.org/jira/browse/HDDS-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877382#comment-16877382
]
Anu Engineer commented on HDDS-1735:
{code:java}
./hadoop-common-project/hadoop-auth/dev-support/findbugsExcludeFile.xml
./hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml
./hadoop-common-project/hadoop-nfs/dev-support/findbugsExcludeFile.xml
./hadoop-common-project/hadoop-minikdc/dev-support/findbugsExcludeFile.xml
./hadoop-common-project/hadoop-kms/dev-support/findbugsExcludeFile.xml
./hadoop-hdds/common/target/findbugsExcludeFile.xml
./hadoop-hdds/common/dev-support/findbugsExcludeFile.xml
./hadoop-hdds/container-service/target/findbugsExcludeFile.xml
./hadoop-hdds/container-service/dev-support/findbugsExcludeFile.xml
./hadoop-hdfs-project/hadoop-hdfs-httpfs/dev-support/findbugsExcludeFile.xml
./hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml
./hadoop-hdfs-project/hadoop-hdfs-nfs/dev-support/findbugsExcludeFile.xml
./hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml
./hadoop-hdfs-project/hadoop-hdfs-rbf/dev-support/findbugsExcludeFile.xml{code}
what are these but suppressions ? Allow me to show you some more examples:
Here is some code from one these files:
{code:java}
{code}
This is code written by someone who made a judgement call, they saw the warning
message from findbugs and since they are experts on this domain made a
judgement call that they know better than a tool. Unless you have evidence to
prove that these lines are wrong; it would be better not to judge these
developers. They contribute work, time and talent to our cause; and you are
saying that grepping few lines over the code allows you to make a judgement
call and deem that this is inferior code? I have shown your claim that Hadoop
does not use suppressions – and the only one was contributed by [~elek] is
incorrect. The above lines prove that suppression is rampant in Hadoop code
base. Much more than Ozone, if you care to look.
What I don't understand is why have such blatant disregard and disrespect for
the community members? if you find a bug, file it as JIRA, please stop making
blanket statements and over-arching pronouncements on code quality. Let us be
specific; if you see an issue; we welcome it; and thank you for your making our
software better; the nature of software development is such that we will have
issues. But let us do it with some consideration; and not say things like I
grepped and found suppressions; therefore your code is bad. if you don't
understand that code; feel free to ask us; perhaps we are wrong; and you have
found a bug; perhaps we have a perfectly valid reason to do it that way.
Once more; I stand by my earlier statement; unless you are willing to measure
and show me numbers – I am going to presume that you have really no really
issues to be solved.
Thank you for the conversation.
> Create separate unit and integration test executor dev-support script
> -
>
> Key: HDDS-1735
> URL: https://issues.apache.org/jira/browse/HDDS-1735
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Attachments: Screen Shot 2019-07-02 at 3.25.33 PM.png
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> hadoop-ozone/dev-support/checks directory contains multiple helper script to
> execute different type of testing (findbugs, rat, unit, build).
> They easily define how tests should be executed, with the following contract:
> * The problems should be printed out to the console
> * in case of test failure a non zero exit code should be used
>
> The tests are working well (in fact I have some experiments with executing
> these scripts on k8s and argo where all the shell scripts are executed
> parallel) but we need some update:
> 1. Most important: the unit tests and integration tests can be separated.
> Integration tests are more flaky and it's better to have a way to run only
> the normal unit tests
> 2. As HDDS-1115 introduced a pom.ozone.xml it's better to use them instead
> of the magical "am pl hadoop-ozone-dist" trick--
> 3. To make it possible to run blockade test in containers we should use - T
> flag with docker-compose
> 4. checkstyle violations are printed out to the console
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1735) Create separate unit and integration test executor dev-support script
[ https://issues.apache.org/jira/browse/HDDS-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877333#comment-16877333 ] Anu Engineer edited comment on HDDS-1735 at 7/3/19 12:13 AM: - Exactly what I predicted. You chickened out. So here is the answer from the Ozone side. I just ran this analysis on HDDS. I have 3 find bug issues, and 2 of them are experimental in Findbugs itself. I am using the latest find bugs. Obviously, you don't dare to do it on Hadoop or Common since your lies will be exposed. Since you so blatantly bad mouth other people and complained about some specific lines in code I am thinking that you did not look at any of the Suppress annotations either. I did; As for the suppressFBWarning, Each one of them was written by a developer making a conscious decision; not by someone running a grep. if you think a specific case is wrong; you should tag that as a Jira, oh – but that requires work :). Plus, I hope you know that we have 22 findbugsExclude files in Hadoop, which is another way of writing these suppressions. Full Disclosure, 8 of them are from Ozone, something which you missed earlier. I will upload my screenshot; if you want to continue this discussion produce the measurements on your side; otherwise I am going to presume that we have nothing material to discuss any further. was (Author: anu): Exactly what I predicted. You chickened out. So here is the answer from the Ozone side. I just ran this analysis on HDDS. I have 3 find bug issues, and 2 of them are experimental in Findbugs itself. I am using the latest find bugs. Obviously, you don't dare to do it on Hadoop or Common since your lies will be exposed. Since you so blatantly bad mouth other people, and do no work; I am expecting that you did not look at any of the Suppress annotations either. I did; As for the suppressFBWarning, Each one of them was written by a developer making a conscious decision; not by someone running a grep. if you think a specific case is wrong; you should tag that as a Jira, oh – but that requires work. Which is not your specialty; but useless trolling is, and if I ask you for a measurement; you will not provide that; since that will again expose how dishonest your comments are. I will upload my screenshot; if you want to continue this discussion; produce the measurements on your side; otherwise I am going to presume that you have been called out on your dishonest/malicious/ignorant comments; I am not sure what the root cause is; > Create separate unit and integration test executor dev-support script > - > > Key: HDDS-1735 > URL: https://issues.apache.org/jira/browse/HDDS-1735 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Attachments: Screen Shot 2019-07-02 at 3.25.33 PM.png > > Time Spent: 1h 10m > Remaining Estimate: 0h > > hadoop-ozone/dev-support/checks directory contains multiple helper script to > execute different type of testing (findbugs, rat, unit, build). > They easily define how tests should be executed, with the following contract: > * The problems should be printed out to the console > * in case of test failure a non zero exit code should be used > > The tests are working well (in fact I have some experiments with executing > these scripts on k8s and argo where all the shell scripts are executed > parallel) but we need some update: > 1. Most important: the unit tests and integration tests can be separated. > Integration tests are more flaky and it's better to have a way to run only > the normal unit tests > 2. As HDDS-1115 introduced a pom.ozone.xml it's better to use them instead > of the magical "am pl hadoop-ozone-dist" trick-- > 3. To make it possible to run blockade test in containers we should use - T > flag with docker-compose > 4. checkstyle violations are printed out to the console -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1731) Implement File CreateFile Request to use Cache and DoubleBuffer
[
https://issues.apache.org/jira/browse/HDDS-1731?focusedWorklogId=271293=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271293
]
ASF GitHub Bot logged work on HDDS-1731:
Author: ASF GitHub Bot
Created on: 03/Jul/19 00:04
Start Date: 03/Jul/19 00:04
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #1044: HDDS-1731.
Implement File CreateFile Request to use Cache and DoubleBuffer.
URL: https://github.com/apache/hadoop/pull/1044#discussion_r299730867
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileCreateRequest.java
##
@@ -0,0 +1,348 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.request.file;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+
+import com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.FileEncryptionInfo;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.container.common.helpers.ExcludeList;
+import org.apache.hadoop.ozone.audit.OMAction;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OMMetrics;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest;
+import org.apache.hadoop.ozone.om.request.key.OMKeyRequest;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.CreateFileRequest;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.KeyArgs;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.OMRequest;
+import org.apache.hadoop.ozone.security.acl.IAccessAuthorizer;
+import org.apache.hadoop.ozone.security.acl.OzoneObj;
+import org.apache.hadoop.util.Time;
+import org.apache.hadoop.utils.UniqueId;
+import org.apache.hadoop.utils.db.Table;
+import org.apache.hadoop.utils.db.TableIterator;
+import org.apache.hadoop.utils.db.cache.CacheKey;
+import org.apache.hadoop.utils.db.cache.CacheValue;
+
+
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.DIRECTORY_EXISTS;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.DIRECTORY_EXISTS_IN_GIVENPATH;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.FILE_EXISTS_IN_GIVENPATH;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.FILE_EXISTS;
+import static
org.apache.hadoop.ozone.om.lock.OzoneManagerLock.Resource.BUCKET_LOCK;
+import static
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.NONE;
+
+/**
+ * Handles create file request.
+ */
+public class OMFileCreateRequest extends OMKeyCreateRequest
+implements OMKeyRequest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OMFileCreateRequest.class);
+ public OMFileCreateRequest(OMRequest omRequest) {
+super(omRequest);
+ }
+
+
+ @Override
+ public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
+CreateFileRequest createFileRequest =
getOmRequest().getCreateFileRequest();
+Preconditions.checkNotNull(createFileRequest);
+
+KeyArgs keyArgs = createFileRequest.getKeyArgs();
+
+if (keyArgs.getKeyName().length() == 0) {
+ // Check if this is the root of the filesystem.
+ // Not throwing exception here, as need to throw exception after
+ // checking volume/bucket exists.
+ return
[jira] [Commented] (HDDS-1735) Create separate unit and integration test executor dev-support script
[
https://issues.apache.org/jira/browse/HDDS-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877368#comment-16877368
]
Eric Yang commented on HDDS-1735:
-
{quote}Exactly what I predicted. You chickened out. So here is the answer from
the Ozone side.{quote}
I am already fixing Hadoop bugs and check style issues daily. Your ask is
going to take a while, and you should know that it takes more than one
individual to build Hadoop. I know that I am contributing my part to make it
better on a daily basis. I only expect you to do the same without resorting to
SuppressFBWarnings to make it worse.
> Create separate unit and integration test executor dev-support script
> -
>
> Key: HDDS-1735
> URL: https://issues.apache.org/jira/browse/HDDS-1735
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Attachments: Screen Shot 2019-07-02 at 3.25.33 PM.png
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> hadoop-ozone/dev-support/checks directory contains multiple helper script to
> execute different type of testing (findbugs, rat, unit, build).
> They easily define how tests should be executed, with the following contract:
> * The problems should be printed out to the console
> * in case of test failure a non zero exit code should be used
>
> The tests are working well (in fact I have some experiments with executing
> these scripts on k8s and argo where all the shell scripts are executed
> parallel) but we need some update:
> 1. Most important: the unit tests and integration tests can be separated.
> Integration tests are more flaky and it's better to have a way to run only
> the normal unit tests
> 2. As HDDS-1115 introduced a pom.ozone.xml it's better to use them instead
> of the magical "am pl hadoop-ozone-dist" trick--
> 3. To make it possible to run blockade test in containers we should use - T
> flag with docker-compose
> 4. checkstyle violations are printed out to the console
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1731) Implement File CreateFile Request to use Cache and DoubleBuffer
[ https://issues.apache.org/jira/browse/HDDS-1731?focusedWorklogId=271282=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271282 ] ASF GitHub Bot logged work on HDDS-1731: Author: ASF GitHub Bot Created on: 02/Jul/19 23:46 Start Date: 02/Jul/19 23:46 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1044: HDDS-1731. Implement File CreateFile Request to use Cache and DoubleBuffer. URL: https://github.com/apache/hadoop/pull/1044#issuecomment-507884383 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 65 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 14 | Maven dependency ordering for branch | | +1 | mvninstall | 470 | trunk passed | | +1 | compile | 247 | trunk passed | | +1 | checkstyle | 70 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 899 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 153 | trunk passed | | 0 | spotbugs | 305 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 489 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 29 | Maven dependency ordering for patch | | +1 | mvninstall | 478 | the patch passed | | +1 | compile | 287 | the patch passed | | +1 | cc | 287 | the patch passed | | +1 | javac | 287 | the patch passed | | +1 | checkstyle | 82 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 721 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 156 | the patch passed | | +1 | findbugs | 619 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 342 | hadoop-hdds in the patch passed. | | -1 | unit | 1643 | hadoop-ozone in the patch failed. | | +1 | asflicense | 42 | The patch does not generate ASF License warnings. | | | | 6985 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestWatchForCommit | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException | | | hadoop.ozone.client.rpc.TestBCSID | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.web.client.TestKeysRatis | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=18.09.5 Server=18.09.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1044/9/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1044 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux 64811b693396 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 8b0d1ad | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1044/9/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1044/9/testReport/ | | Max. process+thread count | 5293 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/common hadoop-ozone/ozone-manager hadoop-ozone/integration-test U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1044/9/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271282) Time Spent: 3h (was: 2h 50m) > Implement File CreateFile Request to use Cache and DoubleBuffer > --- > > Key: HDDS-1731 > URL: https://issues.apache.org/jira/browse/HDDS-1731 > Project: Hadoop Distributed Data Store > Issue
[jira] [Commented] (HDDS-1609) Remove hard coded uid from Ozone docker image
[
https://issues.apache.org/jira/browse/HDDS-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877365#comment-16877365
]
Anu Engineer commented on HDDS-1609:
{quote}it is common practice to swap out open ssh daemon with Apache Mina sshd
running with uid 74, or replace ncftp with vsftp with uid 14
{quote}
You can do that, since all these examples of applications which are
semantically equivalent.
As far as I know, there is no equivalent UID in Unix that is similar to Ozone.
So I don't think we can easily make this decision. So unless we have a better
solution; we will live with what we have.
Thank you for your comments though.
> Remove hard coded uid from Ozone docker image
> -
>
> Key: HDDS-1609
> URL: https://issues.apache.org/jira/browse/HDDS-1609
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Eric Yang
>Priority: Major
> Fix For: 0.5.0
>
> Attachments: linux.txt, log.html, osx.txt, report.html
>
>
> Hadoop-runner image is hard coded to [USER
> hadoop|https://github.com/apache/hadoop/blob/docker-hadoop-runner-jdk11/Dockerfile#L45]
> and user hadoop is hard coded to uid 1000. This arrangement complicates
> development environment where host user is different uid from 1000. External
> bind mount locations are written data as uid 1000. This can prevent
> development environment from clean up test data.
> Docker documentation stated that "The best way to prevent
> privilege-escalation attacks from within a container is to configure your
> container’s applications to run as unprivileged users." From Ozone
> architecture point of view, there is no reason to run Ozone daemon to require
> privileged user or hard coded user.
> h3. Solution 1
> It would be best to support running docker container as host user to reduce
> friction. User should be able to run:
> {code}
> docker run -u $(id -u):$(id -g) ...
> {code}
> or in docker-compose file:
> {code}
> user: "${UID}:${GID}"
> {code}
> By doing this, the user will be name less in docker container. Some commands
> may warn that user does not have a name. This can be resolved by mounting
> /etc/passwd or a file that looks like /etc/passwd that contain host user
> entry.
> h3. Solution 2
> Move the hard coded user to range < 200. The default linux profile reserves
> service users < 200 to have umask that keep data private to service user or
> group writable, if service shares group with other service users. Register
> the service user with Linux vendors to ensure that there is a reserved uid
> for Hadoop user or pick one that works for Hadoop. This is a longer route to
> pursuit, and may not be fruitful.
> h3. Solution 3
> Default the docker image to have sssd client installed. This will allow
> docker image to see host level names by binding sssd socket. The instruction
> for doing this is located at in [Hadoop website|
> https://hadoop.apache.org/docs/r3.1.2/hadoop-yarn/hadoop-yarn-site/DockerContainers.html#User_Management_in_Docker_Container].
> The pre-requisites for this approach will require the host level system to
> have sssd installed. For production system, there is a 99% chance that sssd
> is installed.
> We may want to support combined solution of 1 and 3 to be proper.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1757) Use ExecutorService in OzoneManagerStateMachine
[ https://issues.apache.org/jira/browse/HDDS-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1757: - Resolution: Fixed Fix Version/s: 0.5.0 Status: Resolved (was: Patch Available) Thank You [~anu] for the review and commit. > Use ExecutorService in OzoneManagerStateMachine > --- > > Key: HDDS-1757 > URL: https://issues.apache.org/jira/browse/HDDS-1757 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > In the current code in applyTransaction we have > CompletableFuture future = CompletableFuture > .supplyAsync(() -> runCommand(request, trxLogIndex)); We are using > ForkJoin#commonPool. > With the current approach we have 2 issues: > # Thread exhausts when using this common pool. > # Not a good practice of using common pool. Found some issues in our testing > by using similarly in RatisPipelineUtils. > # OM DB's across replica can be out of sync when the apply transactions are > applied in out of order. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1712) Remove sudo access from Ozone docker image
[ https://issues.apache.org/jira/browse/HDDS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDDS-1712: --- Resolution: Won't Fix Status: Resolved (was: Patch Available) Not an issue, CVE is known and addressed. > Remove sudo access from Ozone docker image > -- > > Key: HDDS-1712 > URL: https://issues.apache.org/jira/browse/HDDS-1712 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: HDDS-1712.001.patch > > > Ozone docker image is given unlimited sudo access to hadoop user. This poses > a security risk where host level user uid 1000 can attach a debugger to the > container process to obtain root access. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1757) Use ExecutorService in OzoneManagerStateMachine
[ https://issues.apache.org/jira/browse/HDDS-1757?focusedWorklogId=271271=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271271 ] ASF GitHub Bot logged work on HDDS-1757: Author: ASF GitHub Bot Created on: 02/Jul/19 23:02 Start Date: 02/Jul/19 23:02 Worklog Time Spent: 10m Work Description: anuengineer commented on pull request #1048: HDDS-1757. Use ExecutorService in OzoneManagerStateMachine. URL: https://github.com/apache/hadoop/pull/1048 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271271) Time Spent: 1h 10m (was: 1h) > Use ExecutorService in OzoneManagerStateMachine > --- > > Key: HDDS-1757 > URL: https://issues.apache.org/jira/browse/HDDS-1757 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > In the current code in applyTransaction we have > CompletableFuture future = CompletableFuture > .supplyAsync(() -> runCommand(request, trxLogIndex)); We are using > ForkJoin#commonPool. > With the current approach we have 2 issues: > # Thread exhausts when using this common pool. > # Not a good practice of using common pool. Found some issues in our testing > by using similarly in RatisPipelineUtils. > # OM DB's across replica can be out of sync when the apply transactions are > applied in out of order. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1757) Use ExecutorService in OzoneManagerStateMachine
[ https://issues.apache.org/jira/browse/HDDS-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877351#comment-16877351 ] Hudson commented on HDDS-1757: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16855 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16855/]) HDDS-1757. Use ExecutorService in OzoneManagerStateMachine. (#1048) (aengineer: rev 91cc19722796877f134fd04f60229ac47a1bd6e0) * (edit) hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java > Use ExecutorService in OzoneManagerStateMachine > --- > > Key: HDDS-1757 > URL: https://issues.apache.org/jira/browse/HDDS-1757 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > In the current code in applyTransaction we have > CompletableFuture future = CompletableFuture > .supplyAsync(() -> runCommand(request, trxLogIndex)); We are using > ForkJoin#commonPool. > With the current approach we have 2 issues: > # Thread exhausts when using this common pool. > # Not a good practice of using common pool. Found some issues in our testing > by using similarly in RatisPipelineUtils. > # OM DB's across replica can be out of sync when the apply transactions are > applied in out of order. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1757) Use ExecutorService in OzoneManagerStateMachine
[ https://issues.apache.org/jira/browse/HDDS-1757?focusedWorklogId=271267=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271267 ] ASF GitHub Bot logged work on HDDS-1757: Author: ASF GitHub Bot Created on: 02/Jul/19 22:58 Start Date: 02/Jul/19 22:58 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1048: HDDS-1757. Use ExecutorService in OzoneManagerStateMachine. URL: https://github.com/apache/hadoop/pull/1048#issuecomment-507874783 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 67 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 455 | trunk passed | | +1 | compile | 246 | trunk passed | | +1 | checkstyle | 60 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 790 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 159 | trunk passed | | 0 | spotbugs | 318 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 506 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 428 | the patch passed | | +1 | compile | 249 | the patch passed | | +1 | javac | 249 | the patch passed | | +1 | checkstyle | 80 | the patch passed | | +1 | mvnsite | 1 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 646 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 161 | the patch passed | | +1 | findbugs | 521 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 241 | hadoop-hdds in the patch passed. | | -1 | unit | 1183 | hadoop-ozone in the patch failed. | | +1 | asflicense | 49 | The patch does not generate ASF License warnings. | | | | 6061 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1048/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1048 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 8a5976ede4f7 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 75b1e45 | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1048/3/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1048/3/testReport/ | | Max. process+thread count | 5018 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/ozone-manager U: hadoop-ozone/ozone-manager | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1048/3/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271267) Time Spent: 1h (was: 50m) > Use ExecutorService in OzoneManagerStateMachine > --- > > Key: HDDS-1757 > URL: https://issues.apache.org/jira/browse/HDDS-1757 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > In the current code in
[jira] [Commented] (HDDS-1712) Remove sudo access from Ozone docker image
[ https://issues.apache.org/jira/browse/HDDS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877347#comment-16877347 ] Anu Engineer commented on HDDS-1712: It was a bug that is fixed. Why will I change my code for an already fixed bug in some other products code base? if that is so; Ozone and Hadoop will have to write special code for all known CVE. I am sure you understand how insane that is. So unless you can show me a real issue, I don't want to commit this. > Remove sudo access from Ozone docker image > -- > > Key: HDDS-1712 > URL: https://issues.apache.org/jira/browse/HDDS-1712 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: HDDS-1712.001.patch > > > Ozone docker image is given unlimited sudo access to hadoop user. This poses > a security risk where host level user uid 1000 can attach a debugger to the > container process to obtain root access. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1712) Remove sudo access from Ozone docker image
[ https://issues.apache.org/jira/browse/HDDS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877346#comment-16877346 ] Eric Yang commented on HDDS-1712: - Yes, it is possible, see [CVE-2019-5736|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-5736]. > Remove sudo access from Ozone docker image > -- > > Key: HDDS-1712 > URL: https://issues.apache.org/jira/browse/HDDS-1712 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: HDDS-1712.001.patch > > > Ozone docker image is given unlimited sudo access to hadoop user. This poses > a security risk where host level user uid 1000 can attach a debugger to the > container process to obtain root access. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1554) Create disk tests for fault injection test
[
https://issues.apache.org/jira/browse/HDDS-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877340#comment-16877340
]
Eric Yang edited comment on HDDS-1554 at 7/2/19 10:41 PM:
--
Patch 10 rebase to current trunk. The current usage command is:
{code}
mvn clean verify -Pit,docker-build
{code}
It does not work without docker-build parameter because the default docker
image name apache/ozone:0.5.0-SNAPSHOT doesn't exist. User can force to build
apache/ozone:0.5.0-SNAPSHOT with:
{code}
mvn clean verify -Ddocker.image=apache/ozone:0.5.0-SNAPSHOT -Pit,docker-build
{code}
[~elek] [~arp] I was unsuccessful in communicating for a better default docker
image name for docker-build profile in HDDS-1667. This is the reason that
docker-build profile needs to be passed even when user is not building docker
image.
Patch 10 version is to discuss if we are open to eliminate docker-build passing
by making the docker.image name default to apache/ozone:${project.version}
because snapshot are most likely locally built image. There is no point to
make further distinction between user and apache docker image name prefix when
docker version tag already make the distinction in my view. I am not sure if
you can agree to this point.
The current test cases is same in the issue description with exception that
Read-only test does not fully initialize metadata. I will update a new version
of read-only test to ensure metadata initialization is done before mark volume
as read-only.
was (Author: eyang):
Patch 10 rebase to current trunk. The current usage command is:
{code}
mvn clean verify -Pit,docker-build
{code}
It does not work without docker-build parameter because the default docker
image name apache/ozone:0.5.0-SNAPSHOT doesn't exist. User can force to build
apache/pzone:0.5.0-SNAPSHOT with:
{code}
mvn clean verify -Ddocker.image=apache/ozone:0.5.0-SNAPSHOT -Pit,docker-build
{code}
[~elek] [~arp] I was unsuccessful in communicating for a better default docker
image name for docker-build profile in HDDS-1667. This is the reason that
docker-build profile needs to be passed even when user is not building docker
image.
Patch 10 version is to discuss if we are open to eliminate docker-build passing
by making the docker.image name default to apache/ozone:${project.version}
because snapshot are most likely locally built image. There is no point to
make further distinction between user and apache docker image name prefix when
docker version tag already make the distinction in my view. I am not sure if
you can agree to this point.
The current test cases is same in the issue description with exception that
Read-only test does not fully initialize metadata. I will update a new version
of read-only test to ensure metadata initialization is done before mark volume
as read-only.
> Create disk tests for fault injection test
> --
>
> Key: HDDS-1554
> URL: https://issues.apache.org/jira/browse/HDDS-1554
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: build
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Labels: pull-request-available
> Attachments: HDDS-1554.001.patch, HDDS-1554.002.patch,
> HDDS-1554.003.patch, HDDS-1554.004.patch, HDDS-1554.005.patch,
> HDDS-1554.006.patch, HDDS-1554.007.patch, HDDS-1554.008.patch,
> HDDS-1554.009.patch, HDDS-1554.010.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The current plan for fault injection disk tests are:
> # Scenario 1 - Read/Write test
> ## Run docker-compose to bring up a cluster
> ## Initialize scm and om
> ## Upload data to Ozone cluster
> ## Verify data is correct
> ## Shutdown cluster
> # Scenario 2 - Read/Only test
> ## Repeat Scenario 1
> ## Mount data disk as read only
> ## Try to write data to Ozone cluster
> ## Validate error message is correct
> ## Shutdown cluster
> # Scenario 3 - Corruption test
> ## Repeat Scenario 2
> ## Shutdown cluster
> ## Modify data disk data
> ## Restart cluster
> ## Validate error message for read from corrupted data
> ## Validate error message for write to corrupted volume
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1554) Create disk tests for fault injection test
[
https://issues.apache.org/jira/browse/HDDS-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877340#comment-16877340
]
Eric Yang commented on HDDS-1554:
-
Patch 10 rebase to current trunk. The current usage command is:
{code}
mvn clean verify -Pit,docker-build
{code}
It does not work without docker-build parameter because the default docker
image name apache/ozone:0.5.0-SNAPSHOT doesn't exist. User can force to build
apache/pzone:0.5.0-SNAPSHOT with:
{code}
mvn clean verify -Ddocker.image=apache/ozone:0.5.0-SNAPSHOT -Pit,docker-build
{code}
[~elek] [~arp] I was unsuccessful in communicating for a better default docker
image name for docker-build profile in HDDS-1667. This is the reason that
docker-build profile needs to be passed even when user is not building docker
image.
Patch 10 version is to discuss if we are open to eliminate docker-build passing
by making the docker.image name default to apache/ozone:${project.version}
because snapshot are most likely locally built image. There is no point to
make further distinction between user and apache docker image name prefix when
docker version tag already make the distinction in my view. I am not sure if
you can agree to this point.
The current test cases is same in the issue description with exception that
Read-only test does not fully initialize metadata. I will update a new version
of read-only test to ensure metadata initialization is done before mark volume
as read-only.
> Create disk tests for fault injection test
> --
>
> Key: HDDS-1554
> URL: https://issues.apache.org/jira/browse/HDDS-1554
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: build
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Labels: pull-request-available
> Attachments: HDDS-1554.001.patch, HDDS-1554.002.patch,
> HDDS-1554.003.patch, HDDS-1554.004.patch, HDDS-1554.005.patch,
> HDDS-1554.006.patch, HDDS-1554.007.patch, HDDS-1554.008.patch,
> HDDS-1554.009.patch, HDDS-1554.010.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The current plan for fault injection disk tests are:
> # Scenario 1 - Read/Write test
> ## Run docker-compose to bring up a cluster
> ## Initialize scm and om
> ## Upload data to Ozone cluster
> ## Verify data is correct
> ## Shutdown cluster
> # Scenario 2 - Read/Only test
> ## Repeat Scenario 1
> ## Mount data disk as read only
> ## Try to write data to Ozone cluster
> ## Validate error message is correct
> ## Shutdown cluster
> # Scenario 3 - Corruption test
> ## Repeat Scenario 2
> ## Shutdown cluster
> ## Modify data disk data
> ## Restart cluster
> ## Validate error message for read from corrupted data
> ## Validate error message for write to corrupted volume
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1735) Create separate unit and integration test executor dev-support script
[ https://issues.apache.org/jira/browse/HDDS-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877337#comment-16877337 ] Anu Engineer commented on HDDS-1735: Uploaded the screenshot; since I did measure the stuff. I am walking the talk, would you like to do that? > Create separate unit and integration test executor dev-support script > - > > Key: HDDS-1735 > URL: https://issues.apache.org/jira/browse/HDDS-1735 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Attachments: Screen Shot 2019-07-02 at 3.25.33 PM.png > > Time Spent: 1h 10m > Remaining Estimate: 0h > > hadoop-ozone/dev-support/checks directory contains multiple helper script to > execute different type of testing (findbugs, rat, unit, build). > They easily define how tests should be executed, with the following contract: > * The problems should be printed out to the console > * in case of test failure a non zero exit code should be used > > The tests are working well (in fact I have some experiments with executing > these scripts on k8s and argo where all the shell scripts are executed > parallel) but we need some update: > 1. Most important: the unit tests and integration tests can be separated. > Integration tests are more flaky and it's better to have a way to run only > the normal unit tests > 2. As HDDS-1115 introduced a pom.ozone.xml it's better to use them instead > of the magical "am pl hadoop-ozone-dist" trick-- > 3. To make it possible to run blockade test in containers we should use - T > flag with docker-compose > 4. checkstyle violations are printed out to the console -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1735) Create separate unit and integration test executor dev-support script
[ https://issues.apache.org/jira/browse/HDDS-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877333#comment-16877333 ] Anu Engineer edited comment on HDDS-1735 at 7/2/19 10:34 PM: - Exactly what I predicted. You chickened out. So here is the answer from the Ozone side. I just ran this analysis on HDDS. I have 3 find bug issues, and 2 of them are experimental in Findbugs itself. I am using the latest find bugs. Obviously, you don't dare to do it on Hadoop or Common since your lies will be exposed. Since you so blatantly bad mouth other people, and do no work; I am expecting that you did not look at any of the Suppress annotations either. I did; As for the suppressFBWarning, Each one of them was written by a developer making a conscious decision; not by someone running a grep. if you think a specific case is wrong; you should tag that as a Jira, oh – but that requires work. Which is not your specialty; but useless trolling is, and if I ask you for a measurement; you will not provide that; since that will again expose how dishonest your comments are. I will upload my screenshot; if you want to continue this discussion; produce the measurements on your side; otherwise I am going to presume that you have been called out on your dishonest/malicious/ignorant comments; I am not sure what the root cause is; was (Author: anu): Exactly what I predicted. You chickened out. So here is the answer from the Ozone side. I just ran this analysis on HDDS. I have 3 find bug issues, and 2 of them are experimental in Findbugs itself. I am using the latest find bugs. Obviously, you don't dare to do it on Hadoop or Common since your lies will be exposed. Since you so blatantly bad mouth other people, and do no work; I am expecting that you did not look at any of the Suppress annotations either. I did; As for the suppressFBWarning, Each one of them was written by a developer making a conscious decision; not by someone running a grep. if you think a specific case is wrong; you should tag that as a Jira, oh – but that requires work. Which is not your specialty; but useless trolling is, and if I ask you for a measurement; you will not provide that; since that will again expose who much dishonest your comments are. I will upload my screenshot; if you want to continue this discussion; produce the measurements on your side; otherwise I am going to presume that you have been called out on your dishonest/malicious/ignorant comments; I am not sure what the root cause is; > Create separate unit and integration test executor dev-support script > - > > Key: HDDS-1735 > URL: https://issues.apache.org/jira/browse/HDDS-1735 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Attachments: Screen Shot 2019-07-02 at 3.25.33 PM.png > > Time Spent: 1h 10m > Remaining Estimate: 0h > > hadoop-ozone/dev-support/checks directory contains multiple helper script to > execute different type of testing (findbugs, rat, unit, build). > They easily define how tests should be executed, with the following contract: > * The problems should be printed out to the console > * in case of test failure a non zero exit code should be used > > The tests are working well (in fact I have some experiments with executing > these scripts on k8s and argo where all the shell scripts are executed > parallel) but we need some update: > 1. Most important: the unit tests and integration tests can be separated. > Integration tests are more flaky and it's better to have a way to run only > the normal unit tests > 2. As HDDS-1115 introduced a pom.ozone.xml it's better to use them instead > of the magical "am pl hadoop-ozone-dist" trick-- > 3. To make it possible to run blockade test in containers we should use - T > flag with docker-compose > 4. checkstyle violations are printed out to the console -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1735) Create separate unit and integration test executor dev-support script
[ https://issues.apache.org/jira/browse/HDDS-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877333#comment-16877333 ] Anu Engineer edited comment on HDDS-1735 at 7/2/19 10:34 PM: - Exactly what I predicted. You chickened out. So here is the answer from the Ozone side. I just ran this analysis on HDDS. I have 3 find bug issues, and 2 of them are experimental in Findbugs itself. I am using the latest find bugs. Obviously, you don't dare to do it on Hadoop or Common since your lies will be exposed. Since you so blatantly bad mouth other people, and do no work; I am expecting that you did not look at any of the Suppress annotations either. I did; As for the suppressFBWarning, Each one of them was written by a developer making a conscious decision; not by someone running a grep. if you think a specific case is wrong; you should tag that as a Jira, oh – but that requires work. Which is not your specialty; but useless trolling is, and if I ask you for a measurement; you will not provide that; since that will again expose who much dishonest your comments are. I will upload my screenshot; if you want to continue this discussion; produce the measurements on your side; otherwise I am going to presume that you have been called out on your dishonest/malicious/ignorant comments; I am not sure what the root cause is; was (Author: anu): Exactly what I predicted. You chickened out. So here is the answer from the Ozone side. I just ran this analysis on HDDS. I have 3 find bug issues, and 2 of them are experimental in Findbugs itself. I am using the latest find bugs. Obviously, you don't dare to do it on Hadoop or Common since your lies will be exposed. Since you so blatantly bad mouth other people, and do no work; I am expecting that you did not look at any of the Suppress annotations either. I did; Each one of them was written by a developer making a conscious decision; not by someone running a grep. if you think a specific case is wrong; you should tag that as a Jira, oh – but that requires work. Which is not your specialty; but useless trolling is, and if I ask you for a measurement; you will not provide that; since that will again expose who much dishonest your comments are. I will upload my screenshot; if you want to continue this discussion; produce the measurements on your side; otherwise I am going to presume that you have been called out on your dishonest/malicious/ignorant comments; I am not sure what the root cause is; > Create separate unit and integration test executor dev-support script > - > > Key: HDDS-1735 > URL: https://issues.apache.org/jira/browse/HDDS-1735 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Attachments: Screen Shot 2019-07-02 at 3.25.33 PM.png > > Time Spent: 1h 10m > Remaining Estimate: 0h > > hadoop-ozone/dev-support/checks directory contains multiple helper script to > execute different type of testing (findbugs, rat, unit, build). > They easily define how tests should be executed, with the following contract: > * The problems should be printed out to the console > * in case of test failure a non zero exit code should be used > > The tests are working well (in fact I have some experiments with executing > these scripts on k8s and argo where all the shell scripts are executed > parallel) but we need some update: > 1. Most important: the unit tests and integration tests can be separated. > Integration tests are more flaky and it's better to have a way to run only > the normal unit tests > 2. As HDDS-1115 introduced a pom.ozone.xml it's better to use them instead > of the magical "am pl hadoop-ozone-dist" trick-- > 3. To make it possible to run blockade test in containers we should use - T > flag with docker-compose > 4. checkstyle violations are printed out to the console -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1735) Create separate unit and integration test executor dev-support script
[ https://issues.apache.org/jira/browse/HDDS-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDDS-1735: --- Attachment: Screen Shot 2019-07-02 at 3.25.33 PM.png > Create separate unit and integration test executor dev-support script > - > > Key: HDDS-1735 > URL: https://issues.apache.org/jira/browse/HDDS-1735 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Attachments: Screen Shot 2019-07-02 at 3.25.33 PM.png > > Time Spent: 1h 10m > Remaining Estimate: 0h > > hadoop-ozone/dev-support/checks directory contains multiple helper script to > execute different type of testing (findbugs, rat, unit, build). > They easily define how tests should be executed, with the following contract: > * The problems should be printed out to the console > * in case of test failure a non zero exit code should be used > > The tests are working well (in fact I have some experiments with executing > these scripts on k8s and argo where all the shell scripts are executed > parallel) but we need some update: > 1. Most important: the unit tests and integration tests can be separated. > Integration tests are more flaky and it's better to have a way to run only > the normal unit tests > 2. As HDDS-1115 introduced a pom.ozone.xml it's better to use them instead > of the magical "am pl hadoop-ozone-dist" trick-- > 3. To make it possible to run blockade test in containers we should use - T > flag with docker-compose > 4. checkstyle violations are printed out to the console -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1735) Create separate unit and integration test executor dev-support script
[ https://issues.apache.org/jira/browse/HDDS-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877333#comment-16877333 ] Anu Engineer commented on HDDS-1735: Exactly what I predicted. You chickened out. So here is the answer from the Ozone side. I just ran this analysis on HDDS. I have 3 find bug issues, and 2 of them are experimental in Findbugs itself. I am using the latest find bugs. Obviously, you don't dare to do it on Hadoop or Common since your lies will be exposed. Since you so blatantly bad mouth other people, and do no work; I am expecting that you did not look at any of the Suppress annotations either. I did; Each one of them was written by a developer making a conscious decision; not by someone running a grep. if you think a specific case is wrong; you should tag that as a Jira, oh – but that requires work. Which is not your specialty; but useless trolling is, and if I ask you for a measurement; you will not provide that; since that will again expose who much dishonest your comments are. I will upload my screenshot; if you want to continue this discussion; produce the measurements on your side; otherwise I am going to presume that you have been called out on your dishonest/malicious/ignorant comments; I am not sure what the root cause is; > Create separate unit and integration test executor dev-support script > - > > Key: HDDS-1735 > URL: https://issues.apache.org/jira/browse/HDDS-1735 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Attachments: Screen Shot 2019-07-02 at 3.25.33 PM.png > > Time Spent: 1h 10m > Remaining Estimate: 0h > > hadoop-ozone/dev-support/checks directory contains multiple helper script to > execute different type of testing (findbugs, rat, unit, build). > They easily define how tests should be executed, with the following contract: > * The problems should be printed out to the console > * in case of test failure a non zero exit code should be used > > The tests are working well (in fact I have some experiments with executing > these scripts on k8s and argo where all the shell scripts are executed > parallel) but we need some update: > 1. Most important: the unit tests and integration tests can be separated. > Integration tests are more flaky and it's better to have a way to run only > the normal unit tests > 2. As HDDS-1115 introduced a pom.ozone.xml it's better to use them instead > of the magical "am pl hadoop-ozone-dist" trick-- > 3. To make it possible to run blockade test in containers we should use - T > flag with docker-compose > 4. checkstyle violations are printed out to the console -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1554) Create disk tests for fault injection test
[ https://issues.apache.org/jira/browse/HDDS-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated HDDS-1554: Attachment: HDDS-1554.010.patch > Create disk tests for fault injection test > -- > > Key: HDDS-1554 > URL: https://issues.apache.org/jira/browse/HDDS-1554 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: build >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Labels: pull-request-available > Attachments: HDDS-1554.001.patch, HDDS-1554.002.patch, > HDDS-1554.003.patch, HDDS-1554.004.patch, HDDS-1554.005.patch, > HDDS-1554.006.patch, HDDS-1554.007.patch, HDDS-1554.008.patch, > HDDS-1554.009.patch, HDDS-1554.010.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > The current plan for fault injection disk tests are: > # Scenario 1 - Read/Write test > ## Run docker-compose to bring up a cluster > ## Initialize scm and om > ## Upload data to Ozone cluster > ## Verify data is correct > ## Shutdown cluster > # Scenario 2 - Read/Only test > ## Repeat Scenario 1 > ## Mount data disk as read only > ## Try to write data to Ozone cluster > ## Validate error message is correct > ## Shutdown cluster > # Scenario 3 - Corruption test > ## Repeat Scenario 2 > ## Shutdown cluster > ## Modify data disk data > ## Restart cluster > ## Validate error message for read from corrupted data > ## Validate error message for write to corrupted volume -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1757) Use ExecutorService in OzoneManagerStateMachine
[ https://issues.apache.org/jira/browse/HDDS-1757?focusedWorklogId=271244=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271244 ] ASF GitHub Bot logged work on HDDS-1757: Author: ASF GitHub Bot Created on: 02/Jul/19 22:06 Start Date: 02/Jul/19 22:06 Worklog Time Spent: 10m Work Description: anuengineer commented on issue #1048: HDDS-1757. Use ExecutorService in OzoneManagerStateMachine. URL: https://github.com/apache/hadoop/pull/1048#issuecomment-507862565 Please feel free to commit when all checks are complete. Thanks for the patch. Appreciate it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271244) Time Spent: 50m (was: 40m) > Use ExecutorService in OzoneManagerStateMachine > --- > > Key: HDDS-1757 > URL: https://issues.apache.org/jira/browse/HDDS-1757 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > In the current code in applyTransaction we have > CompletableFuture future = CompletableFuture > .supplyAsync(() -> runCommand(request, trxLogIndex)); We are using > ForkJoin#commonPool. > With the current approach we have 2 issues: > # Thread exhausts when using this common pool. > # Not a good practice of using common pool. Found some issues in our testing > by using similarly in RatisPipelineUtils. > # OM DB's across replica can be out of sync when the apply transactions are > applied in out of order. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1712) Remove sudo access from Ozone docker image
[ https://issues.apache.org/jira/browse/HDDS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877325#comment-16877325 ] Anu Engineer commented on HDDS-1712: So are you saying that some being root inside a container, becomes root on the base node ? is that what you are claiming ? > Remove sudo access from Ozone docker image > -- > > Key: HDDS-1712 > URL: https://issues.apache.org/jira/browse/HDDS-1712 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: HDDS-1712.001.patch > > > Ozone docker image is given unlimited sudo access to hadoop user. This poses > a security risk where host level user uid 1000 can attach a debugger to the > container process to obtain root access. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1712) Remove sudo access from Ozone docker image
[
https://issues.apache.org/jira/browse/HDDS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877318#comment-16877318
]
Eric Yang commented on HDDS-1712:
-
{quote}Where is the sudo added? in the root node or inside the container?
{quote}
In the container.
The patch applies to remotes/origin/runner-latest branch in
https://github.com/apache/hadoop-docker-ozone repository.
> Remove sudo access from Ozone docker image
> --
>
> Key: HDDS-1712
> URL: https://issues.apache.org/jira/browse/HDDS-1712
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: HDDS-1712.001.patch
>
>
> Ozone docker image is given unlimited sudo access to hadoop user. This poses
> a security risk where host level user uid 1000 can attach a debugger to the
> container process to obtain root access.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1731) Implement File CreateFile Request to use Cache and DoubleBuffer
[ https://issues.apache.org/jira/browse/HDDS-1731?focusedWorklogId=271240=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271240 ] ASF GitHub Bot logged work on HDDS-1731: Author: ASF GitHub Bot Created on: 02/Jul/19 21:49 Start Date: 02/Jul/19 21:49 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #1044: HDDS-1731. Implement File CreateFile Request to use Cache and DoubleBuffer. URL: https://github.com/apache/hadoop/pull/1044#discussion_r299699515 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCreateRequest.java ## @@ -162,73 +164,88 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, KeyArgs keyArgs = createKeyRequest.getKeyArgs(); - String volumeName = keyArgs.getVolumeName(); String bucketName = keyArgs.getBucketName(); String keyName = keyArgs.getKeyName(); OMMetrics omMetrics = ozoneManager.getMetrics(); omMetrics.incNumKeyAllocates(); -AuditLogger auditLogger = ozoneManager.getAuditLogger(); - -Map auditMap = buildKeyArgsAuditMap(keyArgs); - -OMResponse.Builder omResponse = OMResponse.newBuilder().setCmdType( -OzoneManagerProtocolProtos.Type.CreateKey).setStatus( -OzoneManagerProtocolProtos.Status.OK).setSuccess(true); - +OMMetadataManager omMetadataManager = ozoneManager.getMetadataManager(); +OmKeyInfo omKeyInfo = null; +final List< OmKeyLocationInfo > locations = new ArrayList<>(); +FileEncryptionInfo encryptionInfo = null; +IOException exception = null; +boolean acquireLock = false; Review comment: Done. during refactoring things for OMFileCreateRequest, missed acquiring lock. Added that back. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271240) Time Spent: 2h 50m (was: 2h 40m) > Implement File CreateFile Request to use Cache and DoubleBuffer > --- > > Key: HDDS-1731 > URL: https://issues.apache.org/jira/browse/HDDS-1731 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > In this Jira, we shall implement createFile request according to the HA > model, and use cache and double buffer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1712) Remove sudo access from Ozone docker image
[
https://issues.apache.org/jira/browse/HDDS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877314#comment-16877314
]
Hadoop QA commented on HDDS-1712:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 10s{color}
| {color:red} HDDS-1712 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDDS-1712 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12973486/HDDS-1712.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/2747/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.10.0 http://yetus.apache.org |
This message was automatically generated.
> Remove sudo access from Ozone docker image
> --
>
> Key: HDDS-1712
> URL: https://issues.apache.org/jira/browse/HDDS-1712
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: HDDS-1712.001.patch
>
>
> Ozone docker image is given unlimited sudo access to hadoop user. This poses
> a security risk where host level user uid 1000 can attach a debugger to the
> container process to obtain root access.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1712) Remove sudo access from Ozone docker image
[ https://issues.apache.org/jira/browse/HDDS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877313#comment-16877313 ] Anu Engineer commented on HDDS-1712: Where is the sudo added? in the root node or inside the container? > Remove sudo access from Ozone docker image > -- > > Key: HDDS-1712 > URL: https://issues.apache.org/jira/browse/HDDS-1712 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: HDDS-1712.001.patch > > > Ozone docker image is given unlimited sudo access to hadoop user. This poses > a security risk where host level user uid 1000 can attach a debugger to the > container process to obtain root access. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1712) Remove sudo access from Ozone docker image
[ https://issues.apache.org/jira/browse/HDDS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated HDDS-1712: Attachment: (was: HDDS-1712.001.patch) > Remove sudo access from Ozone docker image > -- > > Key: HDDS-1712 > URL: https://issues.apache.org/jira/browse/HDDS-1712 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Eric Yang >Priority: Major > Attachments: HDDS-1712.001.patch > > > Ozone docker image is given unlimited sudo access to hadoop user. This poses > a security risk where host level user uid 1000 can attach a debugger to the > container process to obtain root access. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1712) Remove sudo access from Ozone docker image
[ https://issues.apache.org/jira/browse/HDDS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated HDDS-1712: Attachment: HDDS-1712.001.patch > Remove sudo access from Ozone docker image > -- > > Key: HDDS-1712 > URL: https://issues.apache.org/jira/browse/HDDS-1712 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Eric Yang >Priority: Major > Attachments: HDDS-1712.001.patch > > > Ozone docker image is given unlimited sudo access to hadoop user. This poses > a security risk where host level user uid 1000 can attach a debugger to the > container process to obtain root access. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1712) Remove sudo access from Ozone docker image
[ https://issues.apache.org/jira/browse/HDDS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated HDDS-1712: Assignee: Eric Yang Status: Patch Available (was: Open) > Remove sudo access from Ozone docker image > -- > > Key: HDDS-1712 > URL: https://issues.apache.org/jira/browse/HDDS-1712 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: HDDS-1712.001.patch > > > Ozone docker image is given unlimited sudo access to hadoop user. This poses > a security risk where host level user uid 1000 can attach a debugger to the > container process to obtain root access. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1712) Remove sudo access from Ozone docker image
[ https://issues.apache.org/jira/browse/HDDS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated HDDS-1712: Attachment: HDDS-1712.001.patch > Remove sudo access from Ozone docker image > -- > > Key: HDDS-1712 > URL: https://issues.apache.org/jira/browse/HDDS-1712 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Eric Yang >Priority: Major > Attachments: HDDS-1712.001.patch > > > Ozone docker image is given unlimited sudo access to hadoop user. This poses > a security risk where host level user uid 1000 can attach a debugger to the > container process to obtain root access. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1616) ManagedChannel references are being leaked in while removing RaftGroup
[
https://issues.apache.org/jira/browse/HDDS-1616?focusedWorklogId=271238=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271238
]
ASF GitHub Bot logged work on HDDS-1616:
Author: ASF GitHub Bot
Created on: 02/Jul/19 21:35
Start Date: 02/Jul/19 21:35
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on issue #1039: HDDS-1616.
ManagedChannel references are being leaked in while removing RaftGroup.
Contributed by Mukul Kumar Singh.
URL: https://github.com/apache/hadoop/pull/1039#issuecomment-507853437
Test failures are not related to this patch.
Thank You @mukul1987 for the contribution.
I have committed this to trunk.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 271238)
Time Spent: 50m (was: 40m)
> ManagedChannel references are being leaked in while removing RaftGroup
> --
>
> Key: HDDS-1616
> URL: https://issues.apache.org/jira/browse/HDDS-1616
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0, 0.4.1
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> ManagedChannel references are being leaked in while removing RaftGroup
> {code}
> May 30, 2019 8:12:20 AM
> org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference
> cleanQueue
> SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=1805,
> target=192.168.0.3:49867} was not shutdown properly!!! ~*~*~*
> Make sure to call shutdown()/shutdownNow() and wait until
> awaitTermination() returns true.
> java.lang.RuntimeException: ManagedChannel allocation site
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:103)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:53)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:44)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:411)
> at
> org.apache.ratis.grpc.client.GrpcClientProtocolClient.(GrpcClientProtocolClient.java:118)
> at
> org.apache.ratis.grpc.client.GrpcClientRpc.lambda$new$0(GrpcClientRpc.java:55)
> at
> org.apache.ratis.util.PeerProxyMap$PeerAndProxy.lambda$getProxy$0(PeerProxyMap.java:61)
> at
> org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:202)
> at
> org.apache.ratis.util.PeerProxyMap$PeerAndProxy.getProxy(PeerProxyMap.java:60)
> at org.apache.ratis.util.PeerProxyMap.getProxy(PeerProxyMap.java:107)
> at
> org.apache.ratis.grpc.client.GrpcClientRpc.sendRequest(GrpcClientRpc.java:91)
> at
> org.apache.ratis.client.impl.RaftClientImpl.sendRequest(RaftClientImpl.java:409)
> at
> org.apache.ratis.client.impl.RaftClientImpl.groupRemove(RaftClientImpl.java:281)
> at
> org.apache.hadoop.hdds.scm.pipeline.RatisPipelineUtils.destroyPipeline(RatisPipelineUtils.java:97)
> at
> org.apache.hadoop.hdds.scm.pipeline.PipelineReportHandler.processPipelineReport(PipelineReportHandler.java:100)
> at
> org.apache.hadoop.hdds.scm.pipeline.PipelineReportHandler.onMessage(PipelineReportHandler.java:80)
> at
> org.apache.hadoop.hdds.scm.pipeline.PipelineReportHandler.onMessage(PipelineReportHandler.java:44)
> at
> org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:85)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1616) ManagedChannel references are being leaked in while removing RaftGroup
[
https://issues.apache.org/jira/browse/HDDS-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-1616:
-
Resolution: Fixed
Fix Version/s: 0.4.1
0.5.0
Status: Resolved (was: Patch Available)
> ManagedChannel references are being leaked in while removing RaftGroup
> --
>
> Key: HDDS-1616
> URL: https://issues.apache.org/jira/browse/HDDS-1616
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0, 0.4.1
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> ManagedChannel references are being leaked in while removing RaftGroup
> {code}
> May 30, 2019 8:12:20 AM
> org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference
> cleanQueue
> SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=1805,
> target=192.168.0.3:49867} was not shutdown properly!!! ~*~*~*
> Make sure to call shutdown()/shutdownNow() and wait until
> awaitTermination() returns true.
> java.lang.RuntimeException: ManagedChannel allocation site
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:103)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:53)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:44)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:411)
> at
> org.apache.ratis.grpc.client.GrpcClientProtocolClient.(GrpcClientProtocolClient.java:118)
> at
> org.apache.ratis.grpc.client.GrpcClientRpc.lambda$new$0(GrpcClientRpc.java:55)
> at
> org.apache.ratis.util.PeerProxyMap$PeerAndProxy.lambda$getProxy$0(PeerProxyMap.java:61)
> at
> org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:202)
> at
> org.apache.ratis.util.PeerProxyMap$PeerAndProxy.getProxy(PeerProxyMap.java:60)
> at org.apache.ratis.util.PeerProxyMap.getProxy(PeerProxyMap.java:107)
> at
> org.apache.ratis.grpc.client.GrpcClientRpc.sendRequest(GrpcClientRpc.java:91)
> at
> org.apache.ratis.client.impl.RaftClientImpl.sendRequest(RaftClientImpl.java:409)
> at
> org.apache.ratis.client.impl.RaftClientImpl.groupRemove(RaftClientImpl.java:281)
> at
> org.apache.hadoop.hdds.scm.pipeline.RatisPipelineUtils.destroyPipeline(RatisPipelineUtils.java:97)
> at
> org.apache.hadoop.hdds.scm.pipeline.PipelineReportHandler.processPipelineReport(PipelineReportHandler.java:100)
> at
> org.apache.hadoop.hdds.scm.pipeline.PipelineReportHandler.onMessage(PipelineReportHandler.java:80)
> at
> org.apache.hadoop.hdds.scm.pipeline.PipelineReportHandler.onMessage(PipelineReportHandler.java:44)
> at
> org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:85)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1616) ManagedChannel references are being leaked in while removing RaftGroup
[
https://issues.apache.org/jira/browse/HDDS-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877311#comment-16877311
]
Hudson commented on HDDS-1616:
--
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16854 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16854/])
HDDS-1616. ManagedChannel references are being leaked in while removing
(bharat: rev 8b0d1adf31ca21992ef8bce3e3b5a038421b4edc)
* (edit)
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java
> ManagedChannel references are being leaked in while removing RaftGroup
> --
>
> Key: HDDS-1616
> URL: https://issues.apache.org/jira/browse/HDDS-1616
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> ManagedChannel references are being leaked in while removing RaftGroup
> {code}
> May 30, 2019 8:12:20 AM
> org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference
> cleanQueue
> SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=1805,
> target=192.168.0.3:49867} was not shutdown properly!!! ~*~*~*
> Make sure to call shutdown()/shutdownNow() and wait until
> awaitTermination() returns true.
> java.lang.RuntimeException: ManagedChannel allocation site
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:103)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:53)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:44)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:411)
> at
> org.apache.ratis.grpc.client.GrpcClientProtocolClient.(GrpcClientProtocolClient.java:118)
> at
> org.apache.ratis.grpc.client.GrpcClientRpc.lambda$new$0(GrpcClientRpc.java:55)
> at
> org.apache.ratis.util.PeerProxyMap$PeerAndProxy.lambda$getProxy$0(PeerProxyMap.java:61)
> at
> org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:202)
> at
> org.apache.ratis.util.PeerProxyMap$PeerAndProxy.getProxy(PeerProxyMap.java:60)
> at org.apache.ratis.util.PeerProxyMap.getProxy(PeerProxyMap.java:107)
> at
> org.apache.ratis.grpc.client.GrpcClientRpc.sendRequest(GrpcClientRpc.java:91)
> at
> org.apache.ratis.client.impl.RaftClientImpl.sendRequest(RaftClientImpl.java:409)
> at
> org.apache.ratis.client.impl.RaftClientImpl.groupRemove(RaftClientImpl.java:281)
> at
> org.apache.hadoop.hdds.scm.pipeline.RatisPipelineUtils.destroyPipeline(RatisPipelineUtils.java:97)
> at
> org.apache.hadoop.hdds.scm.pipeline.PipelineReportHandler.processPipelineReport(PipelineReportHandler.java:100)
> at
> org.apache.hadoop.hdds.scm.pipeline.PipelineReportHandler.onMessage(PipelineReportHandler.java:80)
> at
> org.apache.hadoop.hdds.scm.pipeline.PipelineReportHandler.onMessage(PipelineReportHandler.java:44)
> at
> org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:85)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1616) ManagedChannel references are being leaked in while removing RaftGroup
[
https://issues.apache.org/jira/browse/HDDS-1616?focusedWorklogId=271237=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271237
]
ASF GitHub Bot logged work on HDDS-1616:
Author: ASF GitHub Bot
Created on: 02/Jul/19 21:34
Start Date: 02/Jul/19 21:34
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #1039:
HDDS-1616. ManagedChannel references are being leaked in while removing
RaftGroup. Contributed by Mukul Kumar Singh.
URL: https://github.com/apache/hadoop/pull/1039
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 271237)
Time Spent: 40m (was: 0.5h)
> ManagedChannel references are being leaked in while removing RaftGroup
> --
>
> Key: HDDS-1616
> URL: https://issues.apache.org/jira/browse/HDDS-1616
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> ManagedChannel references are being leaked in while removing RaftGroup
> {code}
> May 30, 2019 8:12:20 AM
> org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference
> cleanQueue
> SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=1805,
> target=192.168.0.3:49867} was not shutdown properly!!! ~*~*~*
> Make sure to call shutdown()/shutdownNow() and wait until
> awaitTermination() returns true.
> java.lang.RuntimeException: ManagedChannel allocation site
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:103)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:53)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:44)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:411)
> at
> org.apache.ratis.grpc.client.GrpcClientProtocolClient.(GrpcClientProtocolClient.java:118)
> at
> org.apache.ratis.grpc.client.GrpcClientRpc.lambda$new$0(GrpcClientRpc.java:55)
> at
> org.apache.ratis.util.PeerProxyMap$PeerAndProxy.lambda$getProxy$0(PeerProxyMap.java:61)
> at
> org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:202)
> at
> org.apache.ratis.util.PeerProxyMap$PeerAndProxy.getProxy(PeerProxyMap.java:60)
> at org.apache.ratis.util.PeerProxyMap.getProxy(PeerProxyMap.java:107)
> at
> org.apache.ratis.grpc.client.GrpcClientRpc.sendRequest(GrpcClientRpc.java:91)
> at
> org.apache.ratis.client.impl.RaftClientImpl.sendRequest(RaftClientImpl.java:409)
> at
> org.apache.ratis.client.impl.RaftClientImpl.groupRemove(RaftClientImpl.java:281)
> at
> org.apache.hadoop.hdds.scm.pipeline.RatisPipelineUtils.destroyPipeline(RatisPipelineUtils.java:97)
> at
> org.apache.hadoop.hdds.scm.pipeline.PipelineReportHandler.processPipelineReport(PipelineReportHandler.java:100)
> at
> org.apache.hadoop.hdds.scm.pipeline.PipelineReportHandler.onMessage(PipelineReportHandler.java:80)
> at
> org.apache.hadoop.hdds.scm.pipeline.PipelineReportHandler.onMessage(PipelineReportHandler.java:44)
> at
> org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:85)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14624) When decommissioning a node, log remaining blocks to replicate periodically
[
https://issues.apache.org/jira/browse/HDFS-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877308#comment-16877308
]
Íñigo Goiri commented on HDFS-14624:
We have similar issues internally.
Feel free to propose more logs to add.
Otherwise we can go with [^HDFS-14624.001.patch].
> When decommissioning a node, log remaining blocks to replicate periodically
> ---
>
> Key: HDFS-14624
> URL: https://issues.apache.org/jira/browse/HDFS-14624
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.3.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Attachments: HDFS-14624.001.patch
>
>
> When a node is marked for decommission, there is a monitor thread which runs
> every 30 seconds by default, and checks if the node still has pending blocks
> to be replicated before the node can complete replication.
> There are two existing debug level messages logged in the monitor thread,
> DatanodeAdminManager$Monitor.check(), which log the correct information
> already, first as the pending blocks are replicated:
> {code:java}
> LOG.debug("Node {} still has {} blocks to replicate "
> + "before it is a candidate to finish {}.",
> dn, blocks.size(), dn.getAdminState());{code}
> And then after the initial set of blocks has completed and a rescan happens:
> {code:java}
> LOG.debug("Node {} {} healthy."
> + " It needs to replicate {} more blocks."
> + " {} is still in progress.", dn,
> isHealthy ? "is": "isn't", blocks.size(), dn.getAdminState());{code}
> I would like to propose moving these messages to INFO level so it is easier
> to monitor decommission progress over time from the Namenode log.
> Based on the default settings, this would result in at most 1 log message per
> node being decommissioned every 30 seconds. The reason this is at the most,
> is because the monitor thread stops after checking after 500K blocks and
> therefore in practice it could be as little as 1 log message per 30 seconds,
> even if many DNs are being decommissioned at the same time.
> Note that the namenode webUI does display the above information, but having
> this in the NN logs would allow progress to be tracked more easily.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1757) Use ExecutorService in OzoneManagerStateMachine
[ https://issues.apache.org/jira/browse/HDDS-1757?focusedWorklogId=271232=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271232 ] ASF GitHub Bot logged work on HDDS-1757: Author: ASF GitHub Bot Created on: 02/Jul/19 21:17 Start Date: 02/Jul/19 21:17 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #1048: HDDS-1757. Use ExecutorService in OzoneManagerStateMachine. URL: https://github.com/apache/hadoop/pull/1048#issuecomment-507848143 Thank You @anuengineer for the review. As suggested, used HadoopExecutor API, and also invoked shutdown during the stop. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271232) Time Spent: 40m (was: 0.5h) > Use ExecutorService in OzoneManagerStateMachine > --- > > Key: HDDS-1757 > URL: https://issues.apache.org/jira/browse/HDDS-1757 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > In the current code in applyTransaction we have > CompletableFuture future = CompletableFuture > .supplyAsync(() -> runCommand(request, trxLogIndex)); We are using > ForkJoin#commonPool. > With the current approach we have 2 issues: > # Thread exhausts when using this common pool. > # Not a good practice of using common pool. Found some issues in our testing > by using similarly in RatisPipelineUtils. > # OM DB's across replica can be out of sync when the apply transactions are > applied in out of order. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1731) Implement File CreateFile Request to use Cache and DoubleBuffer
[ https://issues.apache.org/jira/browse/HDDS-1731?focusedWorklogId=271231=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271231 ] ASF GitHub Bot logged work on HDDS-1731: Author: ASF GitHub Bot Created on: 02/Jul/19 21:15 Start Date: 02/Jul/19 21:15 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1044: HDDS-1731. Implement File CreateFile Request to use Cache and DoubleBuffer. URL: https://github.com/apache/hadoop/pull/1044#issuecomment-507847438 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 30 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 12 | Maven dependency ordering for branch | | +1 | mvninstall | 464 | trunk passed | | +1 | compile | 269 | trunk passed | | +1 | checkstyle | 63 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 845 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 150 | trunk passed | | 0 | spotbugs | 348 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 561 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 18 | Maven dependency ordering for patch | | +1 | mvninstall | 473 | the patch passed | | +1 | compile | 271 | the patch passed | | +1 | cc | 271 | the patch passed | | +1 | javac | 271 | the patch passed | | +1 | checkstyle | 66 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 659 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 155 | the patch passed | | +1 | findbugs | 562 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 149 | hadoop-hdds in the patch failed. | | -1 | unit | 1346 | hadoop-ozone in the patch failed. | | +1 | asflicense | 35 | The patch does not generate ASF License warnings. | | | | 6262 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.ozoneimpl.TestOzoneContainer | | | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.client.rpc.TestCommitWatcher | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1044/8/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1044 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux 5dc57384e14b 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 564758a | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1044/8/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1044/8/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1044/8/testReport/ | | Max. process+thread count | 4551 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/common hadoop-ozone/ozone-manager hadoop-ozone/integration-test U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1044/8/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271231) Time Spent: 2h 40m (was: 2.5h) > Implement File CreateFile Request to use Cache and DoubleBuffer > --- > > Key: HDDS-1731 > URL:
[jira] [Work logged] (HDDS-1731) Implement File CreateFile Request to use Cache and DoubleBuffer
[ https://issues.apache.org/jira/browse/HDDS-1731?focusedWorklogId=271228=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271228 ] ASF GitHub Bot logged work on HDDS-1731: Author: ASF GitHub Bot Created on: 02/Jul/19 21:11 Start Date: 02/Jul/19 21:11 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1044: HDDS-1731. Implement File CreateFile Request to use Cache and DoubleBuffer. URL: https://github.com/apache/hadoop/pull/1044#issuecomment-507846189 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 34 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 13 | Maven dependency ordering for branch | | +1 | mvninstall | 474 | trunk passed | | +1 | compile | 253 | trunk passed | | +1 | checkstyle | 60 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 790 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 147 | trunk passed | | 0 | spotbugs | 318 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 505 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 16 | Maven dependency ordering for patch | | +1 | mvninstall | 427 | the patch passed | | +1 | compile | 245 | the patch passed | | +1 | cc | 245 | the patch passed | | +1 | javac | 245 | the patch passed | | +1 | checkstyle | 63 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 608 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 139 | the patch passed | | +1 | findbugs | 532 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 242 | hadoop-hdds in the patch passed. | | -1 | unit | 1722 | hadoop-ozone in the patch failed. | | +1 | asflicense | 51 | The patch does not generate ASF License warnings. | | | | 6461 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.TestMiniChaosOzoneCluster | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.client.rpc.TestReadRetries | | | hadoop.ozone.client.rpc.TestCommitWatcher | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1044/6/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1044 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux 437c028a09ec 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 564758a | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1044/6/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1044/6/testReport/ | | Max. process+thread count | 4511 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/common hadoop-ozone/ozone-manager hadoop-ozone/integration-test U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1044/6/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271228) Time Spent: 2.5h (was: 2h 20m) > Implement File CreateFile Request to use Cache and DoubleBuffer > --- > > Key: HDDS-1731 > URL: https://issues.apache.org/jira/browse/HDDS-1731 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat
[jira] [Commented] (HDFS-14624) When decommissioning a node, log remaining blocks to replicate periodically
[
https://issues.apache.org/jira/browse/HDFS-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877303#comment-16877303
]
Stephen O'Donnell commented on HDFS-14624:
--
This request is driven from requests by our support team and customers, who
state that tracking decommission progress is hard.
The same information is there in JMX, and the namenode WebUI shows the current
snapshot nicely. In support cases we tend to get logs rather than JMX samples,
and it can be useful to see when decommission got stuck, and track the progress
over time to see if it is getting faster or slower etc.
We can do this by adjusting the log level of the DatanodeAdminManager 'on the
fly', but that is often only after the problem occurs, so it would be nice to
have this logged as normal info messages provided they don't create too much
spam.
> When decommissioning a node, log remaining blocks to replicate periodically
> ---
>
> Key: HDFS-14624
> URL: https://issues.apache.org/jira/browse/HDFS-14624
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.3.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Attachments: HDFS-14624.001.patch
>
>
> When a node is marked for decommission, there is a monitor thread which runs
> every 30 seconds by default, and checks if the node still has pending blocks
> to be replicated before the node can complete replication.
> There are two existing debug level messages logged in the monitor thread,
> DatanodeAdminManager$Monitor.check(), which log the correct information
> already, first as the pending blocks are replicated:
> {code:java}
> LOG.debug("Node {} still has {} blocks to replicate "
> + "before it is a candidate to finish {}.",
> dn, blocks.size(), dn.getAdminState());{code}
> And then after the initial set of blocks has completed and a rescan happens:
> {code:java}
> LOG.debug("Node {} {} healthy."
> + " It needs to replicate {} more blocks."
> + " {} is still in progress.", dn,
> isHealthy ? "is": "isn't", blocks.size(), dn.getAdminState());{code}
> I would like to propose moving these messages to INFO level so it is easier
> to monitor decommission progress over time from the Namenode log.
> Based on the default settings, this would result in at most 1 log message per
> node being decommissioned every 30 seconds. The reason this is at the most,
> is because the monitor thread stops after checking after 500K blocks and
> therefore in practice it could be as little as 1 log message per 30 seconds,
> even if many DNs are being decommissioned at the same time.
> Note that the namenode webUI does display the above information, but having
> this in the NN logs would allow progress to be tracked more easily.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1731) Implement File CreateFile Request to use Cache and DoubleBuffer
[ https://issues.apache.org/jira/browse/HDDS-1731?focusedWorklogId=271224=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271224 ] ASF GitHub Bot logged work on HDDS-1731: Author: ASF GitHub Bot Created on: 02/Jul/19 21:07 Start Date: 02/Jul/19 21:07 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1044: HDDS-1731. Implement File CreateFile Request to use Cache and DoubleBuffer. URL: https://github.com/apache/hadoop/pull/1044#issuecomment-507844856 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 36 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 10 | Maven dependency ordering for branch | | +1 | mvninstall | 473 | trunk passed | | +1 | compile | 247 | trunk passed | | +1 | checkstyle | 56 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 778 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 145 | trunk passed | | 0 | spotbugs | 201 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 125 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 19 | Maven dependency ordering for patch | | +1 | mvninstall | 428 | the patch passed | | +1 | compile | 247 | the patch passed | | +1 | cc | 247 | the patch passed | | +1 | javac | 247 | the patch passed | | +1 | checkstyle | 68 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 599 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 148 | the patch passed | | +1 | findbugs | 530 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 238 | hadoop-hdds in the patch passed. | | -1 | unit | 1363 | hadoop-ozone in the patch failed. | | +1 | asflicense | 38 | The patch does not generate ASF License warnings. | | | | 5876 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestReadRetries | | | hadoop.ozone.web.client.TestKeys | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1044/7/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1044 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux 48ffe2696b5f 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 564758a | | Default Java | 1.8.0_212 | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1044/7/artifact/out/branch-findbugs-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1044/7/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1044/7/testReport/ | | Max. process+thread count | 2027 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/common hadoop-ozone/ozone-manager hadoop-ozone/integration-test U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1044/7/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271224) Time Spent: 2h 20m (was: 2h 10m) > Implement File CreateFile Request to use Cache and DoubleBuffer > --- > > Key: HDDS-1731 > URL: https://issues.apache.org/jira/browse/HDDS-1731 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major >
[jira] [Work logged] (HDDS-1757) Use ExecutorService in OzoneManagerStateMachine
[ https://issues.apache.org/jira/browse/HDDS-1757?focusedWorklogId=271222=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271222 ] ASF GitHub Bot logged work on HDDS-1757: Author: ASF GitHub Bot Created on: 02/Jul/19 21:02 Start Date: 02/Jul/19 21:02 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1048: HDDS-1757. Use ExecutorService in OzoneManagerStateMachine. URL: https://github.com/apache/hadoop/pull/1048#issuecomment-507843475 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 68 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 468 | trunk passed | | +1 | compile | 231 | trunk passed | | +1 | checkstyle | 70 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 880 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 157 | trunk passed | | 0 | spotbugs | 305 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 494 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 430 | the patch passed | | +1 | compile | 253 | the patch passed | | +1 | javac | 253 | the patch passed | | +1 | checkstyle | 74 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 730 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 179 | the patch passed | | +1 | findbugs | 605 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 335 | hadoop-hdds in the patch passed. | | -1 | unit | 2108 | hadoop-ozone in the patch failed. | | +1 | asflicense | 52 | The patch does not generate ASF License warnings. | | | | 7316 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.TestMiniOzoneCluster | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException | | | hadoop.ozone.om.TestOzoneManager | | | hadoop.ozone.client.rpc.TestContainerStateMachine | | | hadoop.ozone.client.rpc.TestCommitWatcher | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=18.09.5 Server=18.09.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1048/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1048 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 7ca884e2b65d 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 564758a | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1048/2/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1048/2/testReport/ | | Max. process+thread count | 3769 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/ozone-manager U: hadoop-ozone/ozone-manager | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1048/2/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271222) Time Spent: 0.5h (was: 20m) > Use ExecutorService in OzoneManagerStateMachine > --- > > Key: HDDS-1757 > URL: https://issues.apache.org/jira/browse/HDDS-1757 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority:
[jira] [Updated] (HDFS-14626) Decommission all nodes hosting last block of open file succeeds unexpectedly
[
https://issues.apache.org/jira/browse/HDFS-14626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell updated HDFS-14626:
-
Attachment: test-to-reproduce.patch
> Decommission all nodes hosting last block of open file succeeds unexpectedly
> -
>
> Key: HDFS-14626
> URL: https://issues.apache.org/jira/browse/HDFS-14626
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Attachments: test-to-reproduce.patch
>
>
> I have been investigating scenarios that cause decommission to hang,
> especially around one long standing issue. That is, an open block on the host
> which is being decommissioned can cause the process to never complete.
> Checking the history, there seems to have been at least one change in
> HDFS-5579 which greatly improved the situation, but from reading comments and
> support cases, there still seems to be some scenarios where open blocks on a
> DN host cause the decommission to get stuck.
> No matter what I try, I have not been able to reproduce this, but I think I
> have uncovered another issue that may partly explain why.
> If I do the following, the nodes will decommission without any issues:
> 1. Create a file and write to it so it crosses a block boundary. Then there
> is one complete block and one under construction block. Keep the file open,
> and write a few bytes periodically.
> 2. Now note the nodes which the UC block is currently being written on, and
> decommission them all.
> 3. The decommission should succeed.
> 4. Now attempt to close the open file, and it will fail to close with an
> error like below, probably as decommissioned nodes are not allowed to send
> IBRs:
> {code:java}
> java.io.IOException: Unable to close file because the last block
> BP-646926902-192.168.0.20-1562099323291:blk_1073741827_1003 does not have
> enough number of replicas.
> at
> org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:968)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:911)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:894)
> at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:849)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101){code}
> Interestingly, if you recommission the nodes without restarting them before
> closing the file, it will close OK, and writes to it can continue even once
> decommission has completed.
> I don't think this is expected - ie decommission should not complete on all
> nodes hosting the last UC block of a file?
> From what I have figured out, I don't think UC blocks are considered in the
> DatanodeAdminManager at all. This is because the original list of blocks it
> cares about, are taken from the Datanode block Iterator, which takes them
> from the DatanodeStorageInfo objects attached to the datanode instance. I
> believe UC blocks don't make it into the DatanodeStoreageInfo until after
> they have been completed and an IBR sent, so the decommission logic never
> considers them.
> What troubles me about this explanation, is how did open files previously
> cause decommission to get stuck if it never checks for them, so I suspect I
> am missing something.
> I will attach a patch with a test case that demonstrates this issue. This
> reproduces on trunk and I also tested on CDH 5.8.1, which is based on the 2.6
> branch, but with a lot of backports.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14626) Decommission all nodes hosting last block of open file succeeds unexpectedly
[
https://issues.apache.org/jira/browse/HDFS-14626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell updated HDFS-14626:
-
Description:
I have been investigating scenarios that cause decommission to hang, especially
around one long standing issue. That is, an open block on the host which is
being decommissioned can cause the process to never complete.
Checking the history, there seems to have been at least one change in HDFS-5579
which greatly improved the situation, but from reading comments and support
cases, there still seems to be some scenarios where open blocks on a DN host
cause the decommission to get stuck.
No matter what I try, I have not been able to reproduce this, but I think I
have uncovered another issue that may partly explain why.
If I do the following, the nodes will decommission without any issues:
1. Create a file and write to it so it crosses a block boundary. Then there is
one complete block and one under construction block. Keep the file open, and
write a few bytes periodically.
2. Now note the nodes which the UC block is currently being written on, and
decommission them all.
3. The decommission should succeed.
4. Now attempt to close the open file, and it will fail to close with an error
like below, probably as decommissioned nodes are not allowed to send IBRs:
{code:java}
java.io.IOException: Unable to close file because the last block
BP-646926902-192.168.0.20-1562099323291:blk_1073741827_1003 does not have
enough number of replicas.
at
org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:968)
at
org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:911)
at
org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:894)
at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:849)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101){code}
Interestingly, if you recommission the nodes without restarting them before
closing the file, it will close OK, and writes to it can continue even once
decommission has completed.
I don't think this is expected - ie decommission should not complete on all
nodes hosting the last UC block of a file?
>From what I have figured out, I don't think UC blocks are considered in the
>DatanodeAdminManager at all. This is because the original list of blocks it
>cares about, are taken from the Datanode block Iterator, which takes them from
>the DatanodeStorageInfo objects attached to the datanode instance. I believe
>UC blocks don't make it into the DatanodeStoreageInfo until after they have
>been completed and an IBR sent, so the decommission logic never considers them.
What troubles me about this explanation, is how did open files previously cause
decommission to get stuck if it never checks for them, so I suspect I am
missing something.
I will attach a patch with a test case that demonstrates this issue. This
reproduces on trunk and I also tested on CDH 5.8.1, which is based on the 2.6
branch, but with a lot of backports.
was:
I have been investigating scenarios that cause decommission to hang, especially
around one long standing issue. That is, an open block on the host which is
being decommissioned can cause the process to never complete.
Checking the history, there seems to have been at least one change in HDFS-5579
which greatly improved the situation, but from reading comments and support
cases, there still seems to be some scenarios where open blocks on a DN host
cause the decommission to get stuck.
No matter what I try, I have not been able to reproduce this, but I think I
have uncovered another issue that may partly explain why.
If I do the following, the nodes will decommission without any issues:
1. Create a file and write to it so it crosses a block boundary. Then there is
one complete block and one under construction block. Keep the file open, and
write a few bytes periodically.
2. Now note the nodes which the UC block is currently being written on, and
decommission them all.
3. The decommission should succeed.
4. Now attempt to close the open file, and it will fail to close with an error
like below, probably as decommissioned nodes are not allowed to send IBRs:
{code}
java.io.IOException: Unable to close file because the last block
BP-646926902-192.168.0.20-1562099323291:blk_1073741827_1003 does not have
enough number of replicas.
at
org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:968)
at
org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:911)
at
org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:894)
at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:849)
at
[jira] [Created] (HDFS-14626) Decommission all nodes hosting last block of open file succeeds unexpectedly
Stephen O'Donnell created HDFS-14626:
Summary: Decommission all nodes hosting last block of open file
succeeds unexpectedly
Key: HDFS-14626
URL: https://issues.apache.org/jira/browse/HDFS-14626
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.3.0
Reporter: Stephen O'Donnell
Assignee: Stephen O'Donnell
I have been investigating scenarios that cause decommission to hang, especially
around one long standing issue. That is, an open block on the host which is
being decommissioned can cause the process to never complete.
Checking the history, there seems to have been at least one change in HDFS-5579
which greatly improved the situation, but from reading comments and support
cases, there still seems to be some scenarios where open blocks on a DN host
cause the decommission to get stuck.
No matter what I try, I have not been able to reproduce this, but I think I
have uncovered another issue that may partly explain why.
If I do the following, the nodes will decommission without any issues:
1. Create a file and write to it so it crosses a block boundary. Then there is
one complete block and one under construction block. Keep the file open, and
write a few bytes periodically.
2. Now note the nodes which the UC block is currently being written on, and
decommission them all.
3. The decommission should succeed.
4. Now attempt to close the open file, and it will fail to close with an error
like below, probably as decommissioned nodes are not allowed to send IBRs:
{code}
java.io.IOException: Unable to close file because the last block
BP-646926902-192.168.0.20-1562099323291:blk_1073741827_1003 does not have
enough number of replicas.
at
org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:968)
at
org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:911)
at
org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:894)
at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:849)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
\{code}
Interestingly, if you recommission the nodes without restarting them before
closing the file, it will close OK, and writes to it can continue even once
decommission has completed.
I don't think this is expected - ie decommission should not complete on all
nodes hosting the last UC block of a file?
>From what I have figured out, I don't think UC blocks are considered in the
>DatanodeAdminManager at all. This is because the original list of blocks it
>cares about, are taken from the Datanode block Iterator, which takes them from
>the DatanodeStorageInfo objects attached to the datanode instance. I believe
>UC blocks don't make it into the DatanodeStoreageInfo until after they have
>been completed and an IBR sent, so the decommission logic never considers them.
What troubles me about this explanation, is how did open files previously cause
decommission to get stuck if it never checks for them, so I suspect I am
missing something.
I will attach a patch with a test case that demonstrates this issue. This
reproduces on trunk and I also tested on CDH 5.8.1, which is based on the 2.6
branch, but with a lot of backports.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1757) Use ExecutorService in OzoneManagerStateMachine
[ https://issues.apache.org/jira/browse/HDDS-1757?focusedWorklogId=271203=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271203 ] ASF GitHub Bot logged work on HDDS-1757: Author: ASF GitHub Bot Created on: 02/Jul/19 20:54 Start Date: 02/Jul/19 20:54 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1048: HDDS-1757. Use ExecutorService in OzoneManagerStateMachine. URL: https://github.com/apache/hadoop/pull/1048#issuecomment-507840773 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 75 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 489 | trunk passed | | +1 | compile | 263 | trunk passed | | +1 | checkstyle | 69 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 954 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 168 | trunk passed | | 0 | spotbugs | 355 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 581 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 472 | the patch passed | | +1 | compile | 274 | the patch passed | | +1 | javac | 274 | the patch passed | | +1 | checkstyle | 80 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 746 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 167 | the patch passed | | +1 | findbugs | 584 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 320 | hadoop-hdds in the patch passed. | | -1 | unit | 1472 | hadoop-ozone in the patch failed. | | +1 | asflicense | 47 | The patch does not generate ASF License warnings. | | | | 6929 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.client.rpc.TestBCSID | | | hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerByPipeline | | | hadoop.ozone.client.rpc.TestCommitWatcher | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=18.09.5 Server=18.09.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1048/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1048 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 0a50f47d0fd3 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 564758a | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1048/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1048/1/testReport/ | | Max. process+thread count | 5284 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/ozone-manager U: hadoop-ozone/ozone-manager | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1048/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271203) Time Spent: 20m (was: 10m) > Use ExecutorService in OzoneManagerStateMachine > --- > > Key: HDDS-1757 > URL: https://issues.apache.org/jira/browse/HDDS-1757 > Project: Hadoop Distributed Data Store > Issue Type:
[jira] [Work logged] (HDDS-1716) Smoketest results are generated with an internal user
[ https://issues.apache.org/jira/browse/HDDS-1716?focusedWorklogId=271202=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271202 ] ASF GitHub Bot logged work on HDDS-1716: Author: ASF GitHub Bot Created on: 02/Jul/19 20:53 Start Date: 02/Jul/19 20:53 Worklog Time Spent: 10m Work Description: elek commented on pull request #1002: HDDS-1716. Smoketest results are generated with an internal user URL: https://github.com/apache/hadoop/pull/1002 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271202) Time Spent: 1h 20m (was: 1h 10m) > Smoketest results are generated with an internal user > - > > Key: HDDS-1716 > URL: https://issues.apache.org/jira/browse/HDDS-1716 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > [~eyang] reported the problem in HDDS-1609 that the smoketest results are > generated a user (the user inside the docker container) which can be > different from the host user. > There is a minimal risk that the test results can be deleted/corrupted by an > other users if the current user is different from uid=1000 > I opened this issue because [~eyang] said me during an offline discussion > that HDDS-1609 is a more complex issue and not only about the ownership of > the test results. > I suggest to handle the two problems in different way. With this patch, the > permission of the test result files can be fixed easily. > In HDDS-1609 we can discuss about general security problems and try to find > generic solution for them. > Steps to reproduce _this_ problem: > # Use a user which is different from uid=1000 > # Create a new ozone build (mvn clean install -f pom.ozone.xml -DskipTests) > # Go to a compose directory (cd > hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/) > # Execute tests (./test.sh) > # check the ownership of the results (ls -lah ./results) > Current result: the owner of the result files are the user uid=1000 > Expected result: the owner of the files should be always the current user > (even if the current uid is different) > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1716) Smoketest results are generated with an internal user
[ https://issues.apache.org/jira/browse/HDDS-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877298#comment-16877298 ] Hudson commented on HDDS-1716: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16853 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16853/]) HDDS-1716. Smoketest results are generated with an internal user (elek: rev 75b1e458b1a9fe48b27fdbe5422e4abb4a20b200) * (edit) hadoop-ozone/dist/src/main/compose/testlib.sh * (edit) hadoop-ozone/dist/src/main/compose/test-all.sh > Smoketest results are generated with an internal user > - > > Key: HDDS-1716 > URL: https://issues.apache.org/jira/browse/HDDS-1716 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > [~eyang] reported the problem in HDDS-1609 that the smoketest results are > generated a user (the user inside the docker container) which can be > different from the host user. > There is a minimal risk that the test results can be deleted/corrupted by an > other users if the current user is different from uid=1000 > I opened this issue because [~eyang] said me during an offline discussion > that HDDS-1609 is a more complex issue and not only about the ownership of > the test results. > I suggest to handle the two problems in different way. With this patch, the > permission of the test result files can be fixed easily. > In HDDS-1609 we can discuss about general security problems and try to find > generic solution for them. > Steps to reproduce _this_ problem: > # Use a user which is different from uid=1000 > # Create a new ozone build (mvn clean install -f pom.ozone.xml -DskipTests) > # Go to a compose directory (cd > hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/) > # Execute tests (./test.sh) > # check the ownership of the results (ls -lah ./results) > Current result: the owner of the result files are the user uid=1000 > Expected result: the owner of the files should be always the current user > (even if the current uid is different) > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1716) Smoketest results are generated with an internal user
[
https://issues.apache.org/jira/browse/HDDS-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDDS-1716:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
{quote}+1 on second attempt of PR #1002.
{quote}
Thanks the review [~eyang]. I am merging it right now.
> Smoketest results are generated with an internal user
> -
>
> Key: HDDS-1716
> URL: https://issues.apache.org/jira/browse/HDDS-1716
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> [~eyang] reported the problem in HDDS-1609 that the smoketest results are
> generated a user (the user inside the docker container) which can be
> different from the host user.
> There is a minimal risk that the test results can be deleted/corrupted by an
> other users if the current user is different from uid=1000
> I opened this issue because [~eyang] said me during an offline discussion
> that HDDS-1609 is a more complex issue and not only about the ownership of
> the test results.
> I suggest to handle the two problems in different way. With this patch, the
> permission of the test result files can be fixed easily.
> In HDDS-1609 we can discuss about general security problems and try to find
> generic solution for them.
> Steps to reproduce _this_ problem:
> # Use a user which is different from uid=1000
> # Create a new ozone build (mvn clean install -f pom.ozone.xml -DskipTests)
> # Go to a compose directory (cd
> hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/)
> # Execute tests (./test.sh)
> # check the ownership of the results (ls -lah ./results)
> Current result: the owner of the result files are the user uid=1000
> Expected result: the owner of the files should be always the current user
> (even if the current uid is different)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1741) Fix prometheus configuration in ozoneperf example cluster
[ https://issues.apache.org/jira/browse/HDDS-1741?focusedWorklogId=271186=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271186 ] ASF GitHub Bot logged work on HDDS-1741: Author: ASF GitHub Bot Created on: 02/Jul/19 20:31 Start Date: 02/Jul/19 20:31 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1045: HDDS-1741 Fix prometheus configuration in ozoneperf example cluster URL: https://github.com/apache/hadoop/pull/1045#issuecomment-507833256 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 73 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | 0 | yamllint | 1 | yamllint was not available. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 609 | trunk passed | | +1 | compile | 252 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 1668 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 154 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 447 | the patch passed | | +1 | compile | 268 | the patch passed | | +1 | javac | 268 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 683 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 152 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 285 | hadoop-hdds in the patch failed. | | -1 | unit | 2032 | hadoop-ozone in the patch failed. | | +1 | asflicense | 46 | The patch does not generate ASF License warnings. | | | | 6001 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdds.scm.container.placement.algorithms.TestSCMContainerPlacementRackAware | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.client.rpc.TestFailureHandlingByClient | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.client.rpc.TestCommitWatcher | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1045/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1045 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient yamllint | | uname | Linux c59f15ace3e1 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 564758a | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1045/2/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1045/2/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1045/2/testReport/ | | Max. process+thread count | 5057 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/dist U: hadoop-ozone/dist | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1045/2/console | | versions | git=2.7.4 maven=3.3.9 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271186) Time Spent: 40m (was: 0.5h) > Fix prometheus configuration in ozoneperf example cluster > - > > Key: HDDS-1741 > URL: https://issues.apache.org/jira/browse/HDDS-1741 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: docker >Affects Versions: 0.4.0 >Reporter: Elek, Marton >Assignee: Istvan Fajth >Priority:
[jira] [Commented] (HDFS-14624) When decommissioning a node, log remaining blocks to replicate periodically
[
https://issues.apache.org/jira/browse/HDFS-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877291#comment-16877291
]
Íñigo Goiri commented on HDFS-14624:
Thanks [~sodonnell] for the details.
I guess this is not too bad.
Any other insight on why you want to do this?
Let's see if we can add more details here.
Otherwise, I'm fine with [^HDFS-14624.001.patch].
> When decommissioning a node, log remaining blocks to replicate periodically
> ---
>
> Key: HDFS-14624
> URL: https://issues.apache.org/jira/browse/HDFS-14624
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.3.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Attachments: HDFS-14624.001.patch
>
>
> When a node is marked for decommission, there is a monitor thread which runs
> every 30 seconds by default, and checks if the node still has pending blocks
> to be replicated before the node can complete replication.
> There are two existing debug level messages logged in the monitor thread,
> DatanodeAdminManager$Monitor.check(), which log the correct information
> already, first as the pending blocks are replicated:
> {code:java}
> LOG.debug("Node {} still has {} blocks to replicate "
> + "before it is a candidate to finish {}.",
> dn, blocks.size(), dn.getAdminState());{code}
> And then after the initial set of blocks has completed and a rescan happens:
> {code:java}
> LOG.debug("Node {} {} healthy."
> + " It needs to replicate {} more blocks."
> + " {} is still in progress.", dn,
> isHealthy ? "is": "isn't", blocks.size(), dn.getAdminState());{code}
> I would like to propose moving these messages to INFO level so it is easier
> to monitor decommission progress over time from the Namenode log.
> Based on the default settings, this would result in at most 1 log message per
> node being decommissioned every 30 seconds. The reason this is at the most,
> is because the monitor thread stops after checking after 500K blocks and
> therefore in practice it could be as little as 1 log message per 30 seconds,
> even if many DNs are being decommissioned at the same time.
> Note that the namenode webUI does display the above information, but having
> this in the NN logs would allow progress to be tracked more easily.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1734) Use maven assembly to create ozone tarball image
[
https://issues.apache.org/jira/browse/HDDS-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877283#comment-16877283
]
Eric Yang commented on HDDS-1734:
-
{quote}can you tell me case where we have use case for this ?{quote}
If Dockerfile has older timestamp in source directory, it may not get
regenerated. This may happen if developer copy the file from another location
which he actively doing development from or the file is shared by another
developer. He can't run mvn clean because he would have to build the full
Ozone source to get some of the jar dependencies.
{quote}Until this is fixed in Hadoop, I am not comfortable doing this for Ozone
and I am -1; Just being risk-averse since we are focused on getting to a Beta
and GA for Ozone. So let us not do something that has no clear value over the
current way. Your best argument is that we will be the same after committing
this patch. I am not sure we want to do that.{quote}
The goal is to have ability to create tarball and store the tarball in maven
repository. This is good for repeatability. I respect your decision, if you
see no value in this patch.
> Use maven assembly to create ozone tarball image
>
>
> Key: HDDS-1734
> URL: https://issues.apache.org/jira/browse/HDDS-1734
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: HDDS-1734.001.patch, HDDS-1734.002.patch,
> HDDS-1734.003.patch
>
>
> Ozone is using tar stitching to create ozone tarball. This prevents down
> stream project to use Ozone tarball as a dependency. It would be nice to
> create Ozone tarball with maven assembly plugin to have ability to cache
> ozone tarball in maven repository. This ability allows docker build to be a
> separate sub-module and referencing to Ozone tarball. This change can help
> docker development to be more agile without making a full project build.
> Test procedure:
> {code:java}
> mvn -f pom.ozone.xml clean install -DskipTests -DskipShade
> -Dmaven.javadoc.skip -Pdist{code}
> Expected result:
> This will install tarball into:
> {code:java}
> ~/.m2/repository/org/apache/hadoop/hadoop-ozone-dist/0.5.0-SNAPSHOT/hadoop-ozone-dist-0.5.0-SNAPSHOT.tar.gz{code}
> Test procedure 2:
> {code:java}
> mvn -f pom.ozone.xml clean package -DskipTests -DskipShade
> -Dmaven.javadoc.skip -Pdist{code}
>
> Expected result:
> hadoop/hadoop-ozone/dist/target directory contains
> ozone-0.5.0-SNAPSHOT.tar.gz file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1731) Implement File CreateFile Request to use Cache and DoubleBuffer
[ https://issues.apache.org/jira/browse/HDDS-1731?focusedWorklogId=271179=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271179 ] ASF GitHub Bot logged work on HDDS-1731: Author: ASF GitHub Bot Created on: 02/Jul/19 20:12 Start Date: 02/Jul/19 20:12 Worklog Time Spent: 10m Work Description: arp7 commented on pull request #1044: HDDS-1731. Implement File CreateFile Request to use Cache and DoubleBuffer. URL: https://github.com/apache/hadoop/pull/1044#discussion_r299664639 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCreateRequest.java ## @@ -162,73 +164,88 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, KeyArgs keyArgs = createKeyRequest.getKeyArgs(); - String volumeName = keyArgs.getVolumeName(); String bucketName = keyArgs.getBucketName(); String keyName = keyArgs.getKeyName(); OMMetrics omMetrics = ozoneManager.getMetrics(); omMetrics.incNumKeyAllocates(); -AuditLogger auditLogger = ozoneManager.getAuditLogger(); - -Map auditMap = buildKeyArgsAuditMap(keyArgs); - -OMResponse.Builder omResponse = OMResponse.newBuilder().setCmdType( -OzoneManagerProtocolProtos.Type.CreateKey).setStatus( -OzoneManagerProtocolProtos.Status.OK).setSuccess(true); - +OMMetadataManager omMetadataManager = ozoneManager.getMetadataManager(); +OmKeyInfo omKeyInfo = null; +final List< OmKeyLocationInfo > locations = new ArrayList<>(); +FileEncryptionInfo encryptionInfo = null; +IOException exception = null; +boolean acquireLock = false; Review comment: There seems to be a bug here. `acquireLock` is never set to true in any code path. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271179) Time Spent: 2h 10m (was: 2h) > Implement File CreateFile Request to use Cache and DoubleBuffer > --- > > Key: HDDS-1731 > URL: https://issues.apache.org/jira/browse/HDDS-1731 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > In this Jira, we shall implement createFile request according to the HA > model, and use cache and double buffer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
