Re: Improve carbondata CDC performance

2021-03-30 Thread Ajantha Bhat
+1 for this improvement, But as this optimization is dependent on the data. There may be a scenario where after you prune with min max also your dataset size remain almost same as original. Which brings in extra overhead of the new operations added. Do you have plan to add some intelligence or

Re: Support SI at Segment level

2021-03-30 Thread Ajantha Bhat
+1 for this proposal. But the other ongoing requirement ( http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Discussion-Presto-Queries-leveraging-Secondary-Index-td105291.html) is dependent on *isSITableEnabled* so, better to wait for it to finish and redesign on top of it.

Re: [DISCUSSION] Describe complex columns

2021-03-30 Thread Ajantha Bhat
Hi, +1 for this improvement. a) you can also print one line of short information about the parent column when describe column is executed to avoid executing again to know what is parent column type. Example, Describe column decimalcolumn on complexcarbontable; *You can mention that

Re: [DISCUSSION] Support JOIN query with spatial index

2021-03-30 Thread Indhumathi
Hello all, Current design is based on Union of polygons identified from polygon table. Based on discussion with customer, need to change the design to support IN_POLYGON_JOIN in below way. Apply IN_POLYGON udf on each polygon identified from Polygon table and apply aggregation/group by each

Re: [DISCUSSION] Support JOIN query with spatial index

2021-03-30 Thread Ajantha Bhat
Hi, I have some doubts and suggestions for the same. Currently, we support these UDFs --> IN_POLYGON, IN_POLYGON_LIST, IN_POLYLINE_LIST, IN_POLYGON_RANGE_LIST but the user needs to give polygon input manually and as polygon can have many points, it is hard to give manually. So, your requirement

Re: [DISCUSSION] Support alter schema for complex types

2021-03-30 Thread Ajantha Bhat
Hi Akshay, The mail description and document content are not matching. For single-level struct also document says cannot support. So, please list down all the work that need to be done in points and then divide which is supported in phase1 and which is supported in phase 2 clearly in the summary