+1 for this proposal. But the other ongoing requirement ( http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Discussion-Presto-Queries-leveraging-Secondary-Index-td105291.html) is dependent on *isSITableEnabled* so, better to wait for it to finish and redesign on top of it.
Thanks, Ajantha On Tue, Mar 23, 2021 at 1:03 PM Mahesh Raju Somalaraju < maheshraju.o...@gmail.com> wrote: > Hi, > > +1 for the feature. > It will make the query faster. > > 1) With design discussion about the feature(SI to prune as a data frame) > has one property to set. > If the data engine wants to use SI as datamap then need to set. if not > set then it will use plan re-write flow. > > So we have to handle this feature in two cases. Can you please check and > update the design as per this? > > References: > SI to prune as a data frame > > https://docs.google.com/document/d/1VZlRYqydjzBXmZcFLQ4Ty-lK8RQlYVDoEfIId7vOaxk/edit?usp=sharing > > Thanks & Regards > Mahesh Raju Somalaraju > > On Wed, Feb 17, 2021 at 4:05 PM Nihal <nihalnit...@gmail.com> wrote: > > > Hi all, > > > > Currently, if the parent(main) table and SI table don’t have the same > valid > > segments then we disable the SI table. And then from the next query > > onwards, > > we scan and prune only the parent table until we trigger the next load or > > REINDEX command (as these commands will make the parent and SI table > > segments in sync). Because of this, queries take more time to give the > > result when SI is disabled. > > > > To solve this problem we are planning to support SI at the segment level. > > It > > means we will not disable SI if the parent and SI table don’t have the > same > > segments, while we will do the pruning on Si for all valid segments, and > > for > > the rest of the segments, we will do the pruning on main/parent table. > > > > > > At the time of pruning with the main table in TableIndex.prune, if SI > > exists > > for the corresponding filter then all segments which are not present in > the > > SI table will be pruned on the corresponding parent table segment. > > > > Please let me know your thought and input about the same. > > > > Regards > > Nihal kumar ojha > > > > > > > > -- > > Sent from: > > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/ > > >
