Re: [DISCUSSION] Support Compaction for Range Sort

2019-04-04 Thread ManishNalla1994
Hi all, https://issues.apache.org/jira/browse/CARBONDATA-3343 Here is the JIRA link for this feature. Design Doc attached in the JIRA. Thanks and Regards, Manish Nalla -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: [DISCUSSION] Support Compaction for Range Sort

2019-04-04 Thread ManishNalla1994
Hi Xuchuanyin, Thanks for asking the doubt. Consider the following example where we have 2 segments: Seg_0 : R1(0-100), R2(100-200), R3(200-300) Seg_1 : R4(0-50), R5(150-250), R6(250-350) Now the new ranges formed after merging all the overlapping ranges for both the segments will be: R1(0-100

Re: [DISCUSSION] Support Compaction for Range Sort

2019-04-04 Thread xuchuanyin
Hi ManishNalla: """ merging the overlapping intervals and getting new intervals(ranges) out of them """ === What do you mean by saying this? Can you give an example for it. -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: [DISCUSSION] Add new compaction type for compacting delta data file

2019-04-04 Thread xuchuanyin
emm, eliminating delta files to enhance query performance is quite reasonable and compaction is a candidate for it. However I have some questions about this, maybe they will help in your design. Q1: A segment with delta files means there are some UD(update/delete) operations on this segment before

Re: [DISCUSSION] Support Incremental load in datamap and other MV datamap enhancement

2019-04-04 Thread chetdb
1. Will dataloading to MV be supported for all the table properties and load options provided by Carbon. ? 2. Will the newly proposed "alter datamap datamapname compact" command support all compaction types(auto/major/minor/custom) ? 3. Would load to partition table having MV also be supported ?