[ https://issues.apache.org/jira/browse/HDFS-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14107095#comment-14107095 ]
Kihwal Lee commented on HDFS-6914: ---------------------------------- We do this with the fsimage_oiv/oiv_legacy that was added in HDFS-6293. The memory requirement is constantly small regardless of the size of fsimage. Is there any reason not to use it? > Resolve huge memory consumption Issue with OIV processing PB-based fsimages > --------------------------------------------------------------------------- > > Key: HDFS-6914 > URL: https://issues.apache.org/jira/browse/HDFS-6914 > Project: Hadoop HDFS > Issue Type: Bug > Affects Versions: 2.4.1 > Reporter: Hao Chen > Labels: hdfs > Fix For: 2.5.1 > > Attachments: HDFS-6914.patch > > > For better managing and supporting a lot of large hadoop clusters in > production, we internally need to automatically export fsimage to delimited > text files in LSR style and then analyse with hive or pig or build system > metrics for real-time analyzing. > However due to the internal layout changes introduced by the protobuf-based > fsimage, OIV processing program consumes excessive amount of memory. For > example, in order to export the fsimage in size of 8GB, it should have taken > about 85GB memory which is really not reasonable and impacted performance of > other services badly in the same server. > To resolve above problem, I submit this patch which will reduce memory > consumption of OIV LSR processing by 50%. -- This message was sent by Atlassian JIRA (v6.2#6252)