But it will introduce new dependencies. Every day mysql has a lots of warning because hive, so I don't want to use mysql.
在 2020-07-22 14:26:36，"Zhou Kang" <zhoukan...@outlook.com> 写道： >I think it’s a better way to redesign a new metadata store based on relational >DB. > >Other data warehouse use RDBMS as metadata storage, such as Hive us MySQL. > >It is easy to organize data by using RDBMS. > >Maybe it’s a big challenge to swtich the meta storage, But I think the benefit >is great too. > >> 2020年7月22日 上午11:08，Rupeng Wang <wangrup...@live.cn> 写道： >> >> Agreed with Xiaoxiang. I think it's a good proposal. Maintaining system >> availability is important. But we need to verify whether the solution you >> provided is feasible and do more tests. >> >> 在 2020/7/22 10:43，“Xiaoxiang Yu”<hit_la...@126.com 代表 x...@apache.org> 写入: >> >> Hi, >> It is a fancy idea from my side. Since it is a major behavior change >> for Kylin core system and may have impact to all components, please provided >> us your detailed design documentation (better in English ^_^). >> I think you may implement your idea and verify if it is works as you >> expect, if your idea is proved works in your production env, maintainer will >> be happy to learn and view your code change. >> Thank you very much and good luck to you. >> >> >> >> >> >> >> >> >> >> >> -- >> >> Best wishes to you ! >> From ：Xiaoxiang Yu >> >> >> >> >> >> 在 2020-07-22 05:39:49，"chuxiao" <crow...@163.com> 写道： >>> 1.读写锁改成分桶和自旋锁，允许瞬时的脏读。 >>> 2.更新元数据时不再广播自己，即修改元数据的进程。需要靠广播刷新的缓存，在更新操作时同步刷新。 >>> 3.缓存更新细粒度，按最小原子进行更新，考虑记录变更的版本号时间戳 >>> 依次往下修改，直到满足设计目标。 >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> 在 2020-07-22 05:27:24，"chuxiao" <crow...@163.com> 写道： >>>> kylin当前的缓存机制，比较适合cube数几十上百，不频繁更新的场景。 >>>> 当有2千以上的cube，每天更新几百个，metadata上的读写锁，任何修改全更新的广播机制，导致一旦有持续的元数据更新，整个集群响应时间大幅拉长，建模和查询频繁出现超时。 >>>> KYLIN-4169缓解了这个问题，但还不够。 >>>> 我想重新设计缓存功能，设计目标是在单个项目1万个cube，每天新建/删除一千个的场景下，建模和查询不会出现频繁超时，保持系统可用性。 >> >> >
