[ https://issues.apache.org/jira/browse/HDFS-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Allen Wittenauer updated HDFS-6914: ----------------------------------- Labels: BB2015-05-TBR hdfs (was: hdfs) > Resolve huge memory consumption Issue with OIV processing PB-based fsimages > --------------------------------------------------------------------------- > > Key: HDFS-6914 > URL: https://issues.apache.org/jira/browse/HDFS-6914 > Project: Hadoop HDFS > Issue Type: Bug > Affects Versions: 2.4.1 > Reporter: Hao Chen > Labels: BB2015-05-TBR, hdfs > Attachments: HDFS-6914.patch, HDFS-6914.v2.patch > > > For better managing and supporting a lot of large hadoop clusters in > production, we internally need to automatically export fsimage to delimited > text files in LSR style and then analyse with hive or pig or build system > metrics for real-time analyzing. > However due to the internal layout changes introduced by the protobuf-based > fsimage, OIV processing program consumes excessive amount of memory. For > example, in order to export the fsimage in size of 8GB, it should have taken > about 85GB memory which is really not reasonable and impacted performance of > other services badly in the same server. > To resolve above problem, I submit this patch which will reduce memory > consumption of OIV LSR processing by 50%. -- This message was sent by Atlassian JIRA (v6.3.4#6332)