+1 for solution 2 Regards Kumar Vishal
On Thu, 30 Jul 2020 at 3:19 PM, Indhumathi <indhumathi...@gmail.com> wrote: > Hi community, > > Currently, i am working on supporting SI with complex array type. > In order to support it, we must decide, how we can store Array type > in SI, to get better performance. > > Solution 1: > Store Array as complex(ARRAY) type in secondary index table. > > Cons: > Pruning arrays of huge data on SI and maintable will be an overhead > and might not give much performance results. > > Solution 2: > Make Array data as flattened and store it as its child DataType in > secondary > index table, which can provide benefit in some scenarios, compared to > solution 1. > (i have raised a PR with this solution). On first level, only one level of > Array > will be supported. > > And also, with this solution, added support to prune SI on rowId(keeping > position id > till rowId,instead of blockletId), with complex types for better > performance. > > Cons: > With this solution, query having more than one array_contains filter > with expressions like AND, cannot be supported on SI, since the array data > will > flattened in SI. > > Inputs and suggestions for any new solution/ changes in above solution are > most welcomed. > > Regards, > Indhumathi > > > > > -- > Sent from: > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/ >
