[ https://issues.apache.org/jira/browse/KYLIN-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16820980#comment-16820980 ]
XiaoXiang Yu edited comment on KYLIN-3962 at 4/18/19 11:50 AM: --------------------------------------------------------------- If I use Flink streaming to ingest streaming message, and write cuboid data(I think this maybe _+<Dimension Array, MeasureAggregator>+_) to external storage(like hbase or redis), I think it will have some drawback: - Using remote storage instead of local storage will increase the data prepare delay. - Introduce external dependency like flink cluster. - Too heavy pressure on external storage(maybe every entered message will cause a r/w to storage layer). - If we decide to use more cuboid, it will cause too much r/w to storage. If we decide to use less cuboid, most query will hit base cuboid, filter and aggregate will be slower if data in remote (we can use memory cache in receiver). was (Author: hit_lacus): If I use Flink streaming to ingest streaming message, and write cuboid data(I think this maybe _+<Dimension Array, MeasureAggregator>+_) to external storage(like hbase or redis), I think it will have some drawback: - Using remote storage instead of local storage will increase the data prepare delay. - Introduce external dependency like flink cluster. - Too heavy pressure on external storage(maybe every entered message will cause a r/w to storage layer). - If we decide to use more cuboid, it will cause too much r/w to storage. If we decide to use more cuboid, most query will hit base cuboid, filter and aggregate will be slower if data in remote (we can use memory cache in receiver). > Support streaming cubing using Spark Streaming or Flink > ------------------------------------------------------- > > Key: KYLIN-3962 > URL: https://issues.apache.org/jira/browse/KYLIN-3962 > Project: Kylin > Issue Type: Improvement > Reporter: Liu Shaohui > Priority: Major > > KYLIN-3654 introduced the Real-time Streaming, but in my opinion, the arch is > a little too complicated to handle. > As streaming frameworks like spark streaming, flink are widely used in many > companies.Can we use the streaming framework to support real time cubing in > Kylin. > This is just a proposal. More discussion and suggestions are welcomed~ > More details of this proposal will be added later. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)