[jira] [Commented] (HDFS-16147) load fsimage with parallelization and compression
[ https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17860410#comment-17860410 ] Stephen O'Donnell commented on HDFS-16147: -- [~suyeon1222] I have no plans to work on this. > load fsimage with parallelization and compression > - > > Key: HDFS-16147 > URL: https://issues.apache.org/jira/browse/HDFS-16147 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namanode >Affects Versions: 3.3.0 >Reporter: kinit >Priority: Minor > Attachments: HDFS-16147.001.patch, HDFS-16147.002.patch, compress_not > parallel.png, compress_parallel.png, subsection.svg > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16147) load fsimage with parallelization and compression
[ https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17860353#comment-17860353 ] SUYEON LEE commented on HDFS-16147: --- [~mofei], [~sodonnell] Are you going to work on this? > load fsimage with parallelization and compression > - > > Key: HDFS-16147 > URL: https://issues.apache.org/jira/browse/HDFS-16147 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namanode >Affects Versions: 3.3.0 >Reporter: kinit >Priority: Minor > Attachments: HDFS-16147.001.patch, HDFS-16147.002.patch, compress_not > parallel.png, compress_parallel.png, subsection.svg > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16147) load fsimage with parallelization and compression
[ https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17393517#comment-17393517 ] liuyongpan commented on HDFS-16147: --- {color:#33}I am very glad to receive your reply:D. I'll take your advice. I have tested this patch on a large image, see attached figure. {color} > load fsimage with parallelization and compression > - > > Key: HDFS-16147 > URL: https://issues.apache.org/jira/browse/HDFS-16147 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namanode >Affects Versions: 3.3.0 >Reporter: liuyongpan >Priority: Minor > Attachments: HDFS-16147.001.patch, HDFS-16147.002.patch, compress_not > parallel.png, compress_parallel.png, subsection.svg > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16147) load fsimage with parallelization and compression
[ https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17393386#comment-17393386 ] Stephen O'Donnell commented on HDFS-16147: -- With your patch in place I think the output file looks like this: {code} OVERALL_STREAM INODE_SECTION (only in index, not in the data stream) COMPRESSED_INODE_SUB_SECTION COMPRESSED_INODE_SUB_SECTION COMPRESSED_INODE_SUB_SECTION ... EMPTY_COMPRESSED_INODE_SUB_SECTION ... DIR_SECTION (only in index, not in the data stream) COMPRESSED_DIR_SUB_SECTION COMPRESSED_DIR_SUB_SECTION ... EMPTY_COMPRESSED_DIR_SUB_SECTION ... {code} The reason for the empty ones, is because at the end of the INODE section, you call commitSectionAndSubSection() and it closes the sub-section and opens a new compressed stream. Then you immediately close the section, which closes it and opens a new one. I don't think it does any harm, but it would be better if it did not do that, if we can fix it without making the code too complex. Then I think the reason this works, is that if you try to read in parallel, it reads each compressed sub-section. This is fine. When you try to read it serially (ie turn parallel off and load an image or use OIV), it will try to read all the compressed INODE sections using a single stream. I think this is like a series of streams concatenated together, and the decompressor must handle that (concatenated streams) and return the output like it is a single compressed stream. We can probably test this out somehow to be sure. In TestFSImage.testNoParallelSectionsWithCompressionEnabled(..) - could you remove or rename that test and make it load an image with parallel enabled rather than the current test which checks it does not work. Provided my understanding is correct, this patch looks mostly good apart from the two things above, but I will give it a more detailed review in the next day or two. Have you tested this patch on a large image with millions of inodes? > load fsimage with parallelization and compression > - > > Key: HDFS-16147 > URL: https://issues.apache.org/jira/browse/HDFS-16147 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namanode >Affects Versions: 3.3.0 >Reporter: liuyongpan >Priority: Minor > Attachments: HDFS-16147.001.patch, HDFS-16147.002.patch, > subsection.svg > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16147) load fsimage with parallelization and compression
[ https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17393356#comment-17393356 ] Stephen O'Donnell commented on HDFS-16147: -- On further review, most of what I wrote above is wrong! When saving the image, there is a single output stream, but each section is compressed within that stream, each as a separate compressed stream, eg: {code} OVERALL_STREAM COMPRESSED_INODE_SECTION COMPRESSED_DIR_SECTION ... {code} You can see this in the commitSection() method, where the stream is finished(). So this means that when we load a section (not in parallel), it jumps to the start of a compressed section, and reads it in full. This means it is still unknown how you can save a compressed image with sub-sections and load it without parallel. Perhaps a compressed stream can read embedded compressed streams within itself - I am not sure, but I would like to understand how this is working. > load fsimage with parallelization and compression > - > > Key: HDFS-16147 > URL: https://issues.apache.org/jira/browse/HDFS-16147 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namanode >Affects Versions: 3.3.0 >Reporter: liuyongpan >Priority: Minor > Attachments: HDFS-16147.001.patch, HDFS-16147.002.patch, > subsection.svg > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16147) load fsimage with parallelization and compression
[ https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17393310#comment-17393310 ] Stephen O'Donnell commented on HDFS-16147: -- When the image is saved, there is a single stream written out serially. To enable parallel load on the image, index entries are added for the sub-sections as they are written out. This means we have a single stream, with the position of the sub-sections saved. That means, when we load the image, there are two choices: 1. We start at the beginning of a section and open a stream and read the entire section. 2. We open several streams at by jumping to that position and read the given length. When you enabled compression too, this means the entire stream is compressed from end to end as a single compressed stream. I wrongly thought there would be many compressed streams within the image file, and that is why I though OIV etc would have trouble reading this. So it makes sense OIV can read the image serially, and the namenode can also read the image with parallel disabled when compression is on. The surprise to me, is that we can load the image in parallel, as that involves jumping into the compressed stream somewhere in the middle and starting to read, which more compression codecs do not support. It was my belief that gzip does not support this. However, looking at the existing code, before this change, I see that we jump around in the stream already: {code} for (FileSummary.Section s : sections) { channel.position(s.getOffset()); InputStream in = new BufferedInputStream(new LimitInputStream(fin, s.getLength())); in = FSImageUtil.wrapInputStreamForCompression(conf, summary.getCodec(), in); {code} So that must mean the compression codecs are splittable somehow, and they are start decompressing from an random position in the stream. Due to this, if the image is compressed, the existing parallel code can be mostly reused to load the sub-sections within the compressed stream. >From the above, could we allow parallel loading of compressed images by simply >removing the code: {code} -if (loadInParallel) { - if (compressionEnabled) { -LOG.warn("Parallel Image loading and saving is not supported when {}" + -" is set to true. Parallel will be disabled.", -DFSConfigKeys.DFS_IMAGE_COMPRESS_KEY); -loadInParallel = false; - } -} {code} Then let the image save compressed with the sub-sections indexed and try to load it? > load fsimage with parallelization and compression > - > > Key: HDFS-16147 > URL: https://issues.apache.org/jira/browse/HDFS-16147 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namanode >Affects Versions: 3.3.0 >Reporter: liuyongpan >Priority: Minor > Attachments: HDFS-16147.001.patch, HDFS-16147.002.patch, > subsection.svg > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16147) load fsimage with parallelization and compression
[ https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17391605#comment-17391605 ] liuyongpan commented on HDFS-16147: --- 1、HDFS-16147.002.patch fix the error of test unit at org.apache.hadoop.hdfs.server.namenode.TestFSImage.testNoParallelSectionsWithCompressionEnabled 2、Upon careful examination, Oiv can indeed work normally, and I can't explain why it works. 3、If I create a parallel compressed image with this patch, and then try to load it in a NN without this patch and parallel loading disabled, the NN still able to load it. It may be hard to understand, but it's true, and you can check it out. > load fsimage with parallelization and compression > - > > Key: HDFS-16147 > URL: https://issues.apache.org/jira/browse/HDFS-16147 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namanode >Affects Versions: 3.3.0 >Reporter: liuyongpan >Priority: Minor > Attachments: HDFS-16147.001.patch, HDFS-16147.002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16147) load fsimage with parallelization and compression
[ https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17390089#comment-17390089 ] Hadoop QA commented on HDFS-16147: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m 46s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 32m 56s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 29s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 18s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 1s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 25s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 17s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 31s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 22m 7s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are enabled, using SpotBugs. {color} | | {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 3m 21s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 19s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 22s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 22s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 11s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 11s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 53s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 21s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 29s{color} | {color:green}{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 3m 21s{color} | {color:g
[jira] [Commented] (HDFS-16147) load fsimage with parallelization and compression
[ https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17389947#comment-17389947 ] liuyongpan commented on HDFS-16147: --- [~sodonnell] ,thank you very much for your advice. I will check it carefully. > load fsimage with parallelization and compression > - > > Key: HDFS-16147 > URL: https://issues.apache.org/jira/browse/HDFS-16147 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namanode >Affects Versions: 3.3.0 >Reporter: liuyongpan >Priority: Minor > Attachments: HDFS-16147.001.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16147) load fsimage with parallelization and compression
[ https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17389923#comment-17389923 ] Stephen O'Donnell commented on HDFS-16147: -- It is important that OIV can read these images. If you create a parallel compressed image with this patch, and then try to load it in a NN without this patch and parallel loading disabled, is the NN still able to load it? I suspect it won't as there are multiple compressed sections, so it cannot read a single compressed stream from end to end. {quote} It can load 300M Fsimage with compression and parallelization , which can improve 50% loading time. {quote} Is the 50% improvement measured against a compressed single threaded load verses parallel compressed loading? How are the load times between "parallel not compressed " vs "parallel compressed"? > load fsimage with parallelization and compression > - > > Key: HDFS-16147 > URL: https://issues.apache.org/jira/browse/HDFS-16147 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namanode >Affects Versions: 3.3.0 >Reporter: liuyongpan >Priority: Minor > Attachments: HDFS-16147.001.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16147) load fsimage with parallelization and compression
[ https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17389905#comment-17389905 ] liuyongpan commented on HDFS-16147: --- sorry , OIV can't work > load fsimage with parallelization and compression > - > > Key: HDFS-16147 > URL: https://issues.apache.org/jira/browse/HDFS-16147 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namanode >Affects Versions: 3.3.0 >Reporter: liuyongpan >Priority: Minor > Attachments: HDFS-16147.001.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16147) load fsimage with parallelization and compression
[ https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17389888#comment-17389888 ] liuyongpan commented on HDFS-16147: --- [~sodonnell] I tested it in my test environment,and checked your question. 1、OIV can read the image, but can not read the image in parallel, I will do it later. 2、Current logic can read both the old Fsimage and the new Fsimage, I have tested. 3、It can load 300M Fsimage with compression and parallelization , which can improve 50% loading time. > load fsimage with parallelization and compression > - > > Key: HDFS-16147 > URL: https://issues.apache.org/jira/browse/HDFS-16147 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namanode >Affects Versions: 3.3.0 >Reporter: liuyongpan >Priority: Minor > Attachments: HDFS-16147.001.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16147) load fsimage with parallelization and compression
[ https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17389880#comment-17389880 ] Stephen O'Donnell commented on HDFS-16147: -- I only quickly looked at this, but I have a few questions. With this change, will tools like OIV be able to read the image? It looks like there are a series of new compressed sections, and OIV currently does not read the image in parallel - it will just try to read it from the start to the end - will this work? If we have parallel enabled and compressed sub-sections, if we disable parallel for some reason, will the image be readable? Have you been able to benchmark loading a large image in parallel with compression enabled and disabled so we can see if compression makes it faster or slower? > load fsimage with parallelization and compression > - > > Key: HDFS-16147 > URL: https://issues.apache.org/jira/browse/HDFS-16147 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namanode >Affects Versions: 3.3.0 >Reporter: liuyongpan >Priority: Minor > Fix For: 3.3.0 > > Attachments: HDFS-16147.001.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org