[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17389861#comment-17389861 ] liuyongpan commented on HDFS-14617: --- Stephen O'Donnell , I have created the issue HDFS-

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17389857#comment-17389857 ] liuyongpan commented on HDFS-14617: --- Stephen O'Donnell , I have created the issue HDFS

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17389174#comment-17389174 ] liuyongpan commented on HDFS-14617: --- of course > Improve fsimage load time by writing

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17388638#comment-17388638 ] Stephen O'Donnell commented on HDFS-14617: -- [~mofei] Would you like to create a

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17388637#comment-17388637 ] liuyongpan commented on HDFS-14617: --- I finished loading fsimage with both parallelizati

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17388632#comment-17388632 ] liuyongpan commented on HDFS-14617: --- I have implemented parallel and compression at the

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331414#comment-17331414 ] Qi Zhu commented on HDFS-14617: --- cc [~sodonnell] [~hexiaoqiao]  Could you help backport to

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973224#comment-16973224 ] Xiaoqiao He commented on HDFS-14617: Thanks [~Feng Yuan] for your pings. IIRC, it is

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973222#comment-16973222 ] Feng Yuan commented on HDFS-14617: -- Sorry [~sodonnell], i am not very clear at here: {c

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973214#comment-16973214 ] Feng Yuan commented on HDFS-14617: -- Hi [~sodonnell] [~hexiaoqiao] {code:java} if (p.getI

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973126#comment-16973126 ] Stephen O'Donnell commented on HDFS-14617: -- I decided to prevent parallel loadin

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973107#comment-16973107 ] Feng Yuan commented on HDFS-14617: -- Hi xiaoqiao [~hexiaoqiao] Can you tell me why the pa

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16972347#comment-16972347 ] Feng Yuan commented on HDFS-14617: -- [~sodonnell] Ok, i understood it. And i have a ano

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16972323#comment-16972323 ] Stephen O'Donnell commented on HDFS-14617: -- [~Feng Yuan] The progress object (pr

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16972250#comment-16972250 ] Feng Yuan commented on HDFS-14617: -- in loadINodeSectionInParallel: {code:java} n

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16972098#comment-16972098 ] Feng Yuan commented on HDFS-14617: -- org.apache.hadoop.hdfs.server.namenode.FSDirectory#c

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933647#comment-16933647 ] Erik Krogen commented on HDFS-14617: Hi [~sodonnell], thanks for working on this JIRA

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16929389#comment-16929389 ] Hudson commented on HDFS-14617: --- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16915937#comment-16915937 ] Chao Sun commented on HDFS-14617: - hanks [~hexiaoqiao]! will watch that JIRA. > Improve

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16915221#comment-16915221 ] He Xiaoqiao commented on HDFS-14617: [~csun],[~sodonnell],[~jojochuang] HDFS-14771 is

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16914722#comment-16914722 ] Wei-Chiu Chuang commented on HDFS-14617: LGTM thank you Stephen. [~hexiaoqiao] i

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16914389#comment-16914389 ] Stephen O'Donnell commented on HDFS-14617: -- [~jojochuang] I have added a release

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16914350#comment-16914350 ] Stephen O'Donnell commented on HDFS-14617: -- In my testing on the 3.3 branch OIV

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16914179#comment-16914179 ] He Xiaoqiao commented on HDFS-14617: [~csun] Thanks for your follows, I apply this pa

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913928#comment-16913928 ] Chao Sun commented on HDFS-14617: - Thanks [~sodonnell] , [~hexiaoqiao]  and [~jojochuang]

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913819#comment-16913819 ] Hudson commented on HDFS-14617: --- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16908206#comment-16908206 ] Stephen O'Donnell commented on HDFS-14617: -- There has been some good discussion

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16902657#comment-16902657 ] Wei-Chiu Chuang commented on HDFS-14617: {quote} Test with fsimage of over 165M i

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16899581#comment-16899581 ] He Xiaoqiao commented on HDFS-14617: [~jojochuang] Thanks for your check, and sorry f

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16899580#comment-16899580 ] Hadoop QA commented on HDFS-14617: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16899369#comment-16899369 ] Wei-Chiu Chuang commented on HDFS-14617: [~hexiaoqiao] thanks for your input! It

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892886#comment-16892886 ] Stephen O'Donnell commented on HDFS-14617: -- [~ekanth], [~jojochuang] mention you

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891965#comment-16891965 ] Stephen O'Donnell commented on HDFS-14617: -- I pushed a new version of this patch

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890288#comment-16890288 ] Stephen O'Donnell commented on HDFS-14617: -- Thanks for all the comments on this.

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16889167#comment-16889167 ] Hadoop QA commented on HDFS-14617: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16888967#comment-16888967 ] He Xiaoqiao commented on HDFS-14617: Apply this patch to branch-2.7.1, and get a good

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16883608#comment-16883608 ] Hadoop QA commented on HDFS-14617: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16883604#comment-16883604 ] Stephen O'Donnell commented on HDFS-14617: -- This image is about 30GB in size and

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16883532#comment-16883532 ] Daryn Sharp commented on HDFS-14617: I'm definitely not opposed to startup gains beca

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880684#comment-16880684 ] Stephen O'Donnell commented on HDFS-14617: -- The time taken to read and parse the

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880666#comment-16880666 ] Daryn Sharp commented on HDFS-14617: Was asked to take a look at this.  I think this

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16877508#comment-16877508 ] He Xiaoqiao commented on HDFS-14617: Just took a look, thanks a lot for this great wo

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16875166#comment-16875166 ] Stephen O'Donnell commented on HDFS-14617: -- [~jojochuang] I am fairly sure this

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16875161#comment-16875161 ] Hadoop QA commented on HDFS-14617: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16875127#comment-16875127 ] Wei-Chiu Chuang commented on HDFS-14617: Looking good. Really appreciate doing th

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16874990#comment-16874990 ] Stephen O'Donnell commented on HDFS-14617: -- I created a Pull Request with my cod

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16874945#comment-16874945 ] Stephen O'Donnell commented on HDFS-14617: -- The technique I used here is very si

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16874895#comment-16874895 ] He Xiaoqiao commented on HDFS-14617: Thanks [~sodonnell] for filing this JIRA and dis