For this sort of issue, we should think about if there is any other way that can address the same problem without modifying metadata table schema. Because, modifying metadata table schema introduces compatibility issues, such as the upgrade path for existing users.
Benedict, as Samarth and Lucas pointed out, it would be nice if you share more details of exactly where the bottleneck is. That will make the problem clearer and get everyone on the same page. On Tue, Apr 6, 2021 at 6:54 PM Benedict Jin <asdf2...@apache.org> wrote: > > Hi Ben Krug, > > +1 for adding the is_deleted column, and then we can create a timing trigger > to clear these old records. > > Regards, > Benedict Jin > > On 2021/04/06 18:28:45, Ben Krug <ben.k...@imply.io> wrote: > > Oh, that's easier than tombstones. flag is_deleted and update timestamp > > (so it gets pulled again). > > > > On Tue, Apr 6, 2021 at 10:48 AM Tijo Thomas <tijothoma...@gmail.com> wrote: > > > > > Abhishek, > > > Good point. Do we need one more col for storing if it's deleted or not? > > > > > > On Tue, Apr 6, 2021 at 4:32 PM Abhishek Agarwal <abhishek.agar...@imply.io > > > > > > > wrote: > > > > > > > If an entry is deleted from the metadata, how is the coordinator going > > > > to > > > > update its own state? > > > > > > > > On Tue, Apr 6, 2021 at 3:38 PM Itai Yaffe <itai.ya...@gmail.com> wrote: > > > > > > > > > Hey, > > > > > I'm not a Druid developer, so it's quite possible I'm missing many > > > > > considerations here, but from a first glance, I like your offer, as it > > > > > resembles the *tsColumn *in JDBC lookups ( > > > > > > > > > > > > > > > > > https://druid.apache.org/docs/latest/development/extensions-core/lookups-cached-global.html#jdbc-lookup > > > > > ). > > > > > > > > > > Anyway, just my 2 cents. > > > > > > > > > > Thanks! > > > > > Itai > > > > > > > > > > On Tue, Apr 6, 2021 at 6:07 AM Benedict Jin <asdf2...@apache.org> > > > wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > Recently, when the Coordinator in our company's Druid cluster pulls > > > > > > metadata, there is a performance bottleneck. The main reason is the > > > > huge > > > > > > amount of metadata, which leads to a very slow process of scanning > > > the > > > > > full > > > > > > table of metadata storage and deserializing metadata. The size of > > > > > > the > > > > > full > > > > > > metadata has been reduced through TTL, Compaction, Rollup, and etc., > > > > but > > > > > > the effect is not very significant. Therefore, I want to design a > > > > scheme > > > > > > for Coordinator to pull metadata incrementally, that is, each time > > > > > > Coordinator only pulls newly added metadata, so as to reduce the > > > query > > > > > > pressure of metadata storage and the pressure of deserializing > > > > metadata. > > > > > > The general idea is to add a column last_update to the > > > > > > druid_segments > > > > > table > > > > > > to record the update time of each record. Furthermore, when we query > > > > the > > > > > > metadata table, we can add filter conditions for the last_update > > > column > > > > > to > > > > > > avoid full table scan operations. Moreover, whether it is MySQL or > > > > > > PostgreSQL as the metadata storage medium, it can support > > > > > > automatic update of the timestamp field, which is somewhat similar > > > to > > > > > the > > > > > > characteristics of triggers. So, have you encountered this problem > > > > > before? > > > > > > If so, how did you solve it? In addition, do you have any > > > > > > suggestions > > > > or > > > > > > comments on the above incremental acquisition of metadata? Please > > > > > > let > > > > me > > > > > > know, thanks a lot. > > > > > > > > > > > > Regards, > > > > > > Benedict Jin > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org > > > > > > For additional commands, e-mail: dev-h...@druid.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Thanks & Regards > > > Tijo Thomas > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org > For additional commands, e-mail: dev-h...@druid.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org For additional commands, e-mail: dev-h...@druid.apache.org