Here in Uber, we are working on a project on sampling Kafka stream data and user query log to decide whether to index columns and other table config like dictionary encoding. This is very useful for self on-boarding Pinot tables. The main focus now is to add index. But if we can remove index too, there will be helpful too.
On Tue, Feb 11, 2020 at 5:12 PM kishore g <g.kish...@gmail.com> wrote: > It's not trivial for users to know when to add an index. It definitely > improves the experience if they can easily add/remove indexes. Regarding > storage, indexes can be quite big depending on the cardinality for e.g. inv > index on something like memberId is not very useful. > > On Tue, Feb 11, 2020 at 5:07 PM Mayank Shrivastava < > mayanks.apa...@gmail.com> > wrote: > > > Subbu, I think the proposal is to reduce the storage cost, which is why I > > asked how often is that seen (outside LinkedIn). > > > > Sent from my iPhone > > > > > On Feb 11, 2020, at 3:38 PM, Subbu Subramaniam <mcvsu...@apache.org> > > wrote: > > > > > > I believe we do not map the index if it is not sset in the table > > config, so the index is not loaded. > > > > > > The cost on storage exists, but hopefully is a lot smaller than memory. > > > > > > -Subbu > > > > > >> On 2020/02/11 22:37:13, kishore g <g.kish...@gmail.com> wrote: > > >> Currently, we only support adding indices to the pinot segment. If the > > >> index is removed from table config, we dont load it but continue to > > keep it > > >> in the segment. The only drawback with this is additional usage of > disk > > >> space. > > >> > > >> We should enhance the segment preprocessor to have the ability to > > rewrite > > >> the segment. > > >> > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org > > > For additional commands, e-mail: dev-h...@pinot.apache.org > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org > > For additional commands, e-mail: dev-h...@pinot.apache.org > > > > >
