Hi Itai Yaffe, Thank you very much for your support, thank you.
Regards, Benedict Jin On 2021/04/06 10:06:45, Itai Yaffe <itai.ya...@gmail.com> wrote: > Hey, > I'm not a Druid developer, so it's quite possible I'm missing many > considerations here, but from a first glance, I like your offer, as it > resembles the *tsColumn *in JDBC lookups ( > https://druid.apache.org/docs/latest/development/extensions-core/lookups-cached-global.html#jdbc-lookup > ). > > Anyway, just my 2 cents. > > Thanks! > Itai > > On Tue, Apr 6, 2021 at 6:07 AM Benedict Jin <asdf2...@apache.org> wrote: > > > Hi all, > > > > Recently, when the Coordinator in our company's Druid cluster pulls > > metadata, there is a performance bottleneck. The main reason is the huge > > amount of metadata, which leads to a very slow process of scanning the full > > table of metadata storage and deserializing metadata. The size of the full > > metadata has been reduced through TTL, Compaction, Rollup, and etc., but > > the effect is not very significant. Therefore, I want to design a scheme > > for Coordinator to pull metadata incrementally, that is, each time > > Coordinator only pulls newly added metadata, so as to reduce the query > > pressure of metadata storage and the pressure of deserializing metadata. > > The general idea is to add a column last_update to the druid_segments table > > to record the update time of each record. Furthermore, when we query the > > metadata table, we can add filter conditions for the last_update column to > > avoid full table scan operations. Moreover, whether it is MySQL or > > PostgreSQL as the metadata storage medium, it can support > > automatic update of the timestamp field, which is somewhat similar to the > > characteristics of triggers. So, have you encountered this problem before? > > If so, how did you solve it? In addition, do you have any suggestions or > > comments on the above incremental acquisition of metadata? Please let me > > know, thanks a lot. > > > > Regards, > > Benedict Jin > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org > > For additional commands, e-mail: dev-h...@druid.apache.org > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org For additional commands, e-mail: dev-h...@druid.apache.org
