[
https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17393386#comment-17393386
]
Stephen O'Donnell commented on HDFS-16147:
------------------------------------------
With your patch in place I think the output file looks like this:
{code}
OVERALL_STREAM
INODE_SECTION (only in index, not in the data stream)
COMPRESSED_INODE_SUB_SECTION
COMPRESSED_INODE_SUB_SECTION
COMPRESSED_INODE_SUB_SECTION
...
EMPTY_COMPRESSED_INODE_SUB_SECTION
...
DIR_SECTION (only in index, not in the data stream)
COMPRESSED_DIR_SUB_SECTION
COMPRESSED_DIR_SUB_SECTION
...
EMPTY_COMPRESSED_DIR_SUB_SECTION
...
{code}
The reason for the empty ones, is because at the end of the INODE section, you
call commitSectionAndSubSection() and it closes the sub-section and opens a new
compressed stream. Then you immediately close the section, which closes it and
opens a new one. I don't think it does any harm, but it would be better if it
did not do that, if we can fix it without making the code too complex.
Then I think the reason this works, is that if you try to read in parallel, it
reads each compressed sub-section. This is fine.
When you try to read it serially (ie turn parallel off and load an image or use
OIV), it will try to read all the compressed INODE sections using a single
stream. I think this is like a series of streams concatenated together, and the
decompressor must handle that (concatenated streams) and return the output like
it is a single compressed stream. We can probably test this out somehow to be
sure.
In TestFSImage.testNoParallelSectionsWithCompressionEnabled(..) - could you
remove or rename that test and make it load an image with parallel enabled
rather than the current test which checks it does not work.
Provided my understanding is correct, this patch looks mostly good apart from
the two things above, but I will give it a more detailed review in the next day
or two.
Have you tested this patch on a large image with millions of inodes?
> load fsimage with parallelization and compression
> -------------------------------------------------
>
> Key: HDFS-16147
> URL: https://issues.apache.org/jira/browse/HDFS-16147
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namanode
> Affects Versions: 3.3.0
> Reporter: liuyongpan
> Priority: Minor
> Attachments: HDFS-16147.001.patch, HDFS-16147.002.patch,
> subsection.svg
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
