[
https://issues.apache.org/jira/browse/CARBONDATA-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jacky Li resolved CARBONDATA-3637.
----------------------------------
Fix Version/s: 2.0.0
Resolution: Fixed
> Improve insert into performance and decrease memory foot print
> --------------------------------------------------------------
>
> Key: CARBONDATA-3637
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3637
> Project: CarbonData
> Issue Type: Improvement
> Reporter: Ajantha Bhat
> Assignee: Ajantha Bhat
> Priority: Major
> Fix For: 2.0.0
>
> Time Spent: 29h 20m
> Remaining Estimate: 0h
>
> Currently carbondata "insert into" uses the CarbonLoadDataCommand itself.
> Load process has steps like parsing and converter step with bad record
> support.
> Insert into doesn't require these steps as data is already validated and
> converted from source table or dataframe.
> Some identified changes are below.
> 1. Need to refactor and separate load and insert at driver side to skip
> converter step and unify flow for No sort and global sort insert.
> 2. Need to avoid reorder of each row. By changing select dataframe's
> projection order itself during the insert into.
> 3. For carbon to carbon insert, need to provide the ReadSupport and use
> RecordReader (vector reader currently doesn't support ReadSupport) to
> handle null values, time stamp cutoff (direct dictionary) from scanRDD
> result.
> 4. Need to handle insert into partition/non-partition table in local sort,
> global sort, no sort, range columns, compaction flow.
> The final goal is to improve insert performance by keeping only required
> logic and also decrease the memory footprint.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
