[ https://issues.apache.org/jira/browse/CARBONDATA-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
kumar vishal resolved CARBONDATA-2889. -------------------------------------- Resolution: Fixed > Support Decoder based fall back mechanism in Local Dictionary > ------------------------------------------------------------- > > Key: CARBONDATA-2889 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2889 > Project: CarbonData > Issue Type: Sub-task > Reporter: Akash R Nilugal > Assignee: Akash R Nilugal > Priority: Major > Time Spent: 9h 20m > Remaining Estimate: 0h > > Currently, when the fallback is initiated for a column page in case of local > dictionary, we are keeping both encoded data > and actual data in memory and then we form the new column page without > dictionary encoding and then at last we free the Encoded Column Page. > Because of this offheap memory footprint increases. > > We can reduce the offheap memory footprint. This can be done using decoder > based fallback mechanism. > This means, no need to keep the actual data along with encoded data in > encoded column page. We can keep only encoded data and to form a new column > page, get the dictionary data from encoded column page by uncompressing and > using dictionary data get the actual data using local dictionary generator > and put it in new column page created and compress it again and give to > consumer for writing blocklet. > > The above process may slow down the loading, but it will reduces the memory > footprint. So we can give a property which will decide whether to take > current fallback procedure or decoder based fallback mechanism dring fallback -- This message was sent by Atlassian JIRA (v7.6.3#76005)