Re: [Discussion] is it necessary to support SORT_COLUMNS modification

2019-04-09 Thread David CaiQiang 
please check JIRA and find the design doc:
https://issues.apache.org/jira/browse/CARBONDATA-3347



-
Best Regards
David Cai
--
Sent from: 
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: [Discussion] is it necessary to support SORT_COLUMNS modification

2019-03-16 Thread xm_zzc 
Nice feature. Just one suggestion: we can support to convert a specified old
segment by new SORT_COLUMNS, for example: resort table table_name for
segment 0.



--
Sent from: 
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: [Discussion] is it necessary to support SORT_COLUMNS modification

2019-03-13 Thread yaojinguo 
nice feature. I think range_column feature can benefit a lot from this
feature.



--
Sent from: 
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

[Discussion] is it necessary to support SORT_COLUMNS modification

2019-03-13 Thread David CaiQiang 
Hi all,
Let's discuss whether it is necessary to support SORT_COLUMNS
modification.
*Background*
"SORT_COLUMNS" is a table level property, and we can't change it after
creating a table.
*Motivation*
 When we want to optimize the query performance and found that it needs
to change SORT_COLUMNS, Carbon should support changing SORT_COLUMNS.
SORT_COLUMNS just like the main data index and impact the data layout.  At
the same time, we can re-sort old segment data by new SORT_COLUMNS.
 *Modification*
 1. loading data use table level "SORT_COLUMNS" and store it as a
segment level property
 2. query should use segment level property to read data files
 3. only compacting segments with same "SORT_COLUMNS"
 4. convert old segment one by one by new SORT_COLUMNS and refresh
DataMap if needed
 5. show segment command output SORT_COLUMNS  of each segment

Any suggestions and questions? 



-
Best Regards
David Cai
--
Sent from: 
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/