Hi Kevin, Yes, we can improve it. The implementation is closely related to supporting pre-aggregate datamaps on the streaming table which we have already implemented some time ago. And same will be reimplemented for MV datamap soon as well. The implementation allows using of pre-aggregate datamap for non-streaming segments and main table for streaming segments. We update the query plan to do union on both the tables and query only the streaming segments for main table. So even in our case also we can use the same way, we can do the union query of MV table and main table(only non loaded datamap segments) and execute the query. We can definitely consider after we support streaming table for MV datamap.
Regards, Ravindra. On Wed, 17 Jul 2019 at 07:55, kevinjmh <[email protected]> wrote: > currently, datamap in carbon applys to all segments. > The roadmap refers to commands like add/drop segment, and also maybe > something > about incremental loading for MV. For these scenes, it is better to make > datamap can be use on segment level instead of disable the datamap when any > datamap data is not ready for any segment. Also this can make datamap > fail-safe and enhance carbon's stablility. > Maybe we can consider about this also. > > > > > ----- > Regards > Manhua > > > > ---Original--- > From: "Ravindra Pesala"<[email protected]> > Date: Tue, Jul 16, 2019 22:31 PM > To: "dev"<[email protected]>; > Subject: [Discussion] Roadmap for Apache CarbonData 2 > > > Hi Community, > > Three years have passed since the launching of the Apache CarbonData > project, CarbonData has become a popular data management solution for > various scenarios. As new workload like AI and new runtime environment like > the cloud is emerging quickly, I think we are reaching a point that needs > to discuss the future of CarbonData. > > To bring CarbonData to a new level to satisfy those new requirements, Jacky > and I drafted a roadmap for CarbonData 2 in the cwiki website. > - English Version: > > https://cwiki.apache.org/confluence/display/CARBONDATA/Apache+CarbonData+2+Roadmap+Proposal > - Chinese Version: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120737492 > > Please feel free to discuss the roadmap in this thread, and we welcome > every feedback to make CarbonData better. > > Thanks and Regards, > Ravindra. -- Thanks & Regards, Ravi
