Hi gang, I am so glad to know that eBay has a solution for realtime olap on kylin. I have some small question:
1. Is it possible to use Yarn as cluster manager for index task. Coordinator process will set up them at specificed period. Yarn will manage : a) retry these task if some failed b) resource allocation c) log collection 2. As I know, ebay’s New Kylin Streaming Solution use replica Set to ensure that income messages wouldn’t lost if some processes lost. I think replica set is a set of kafka cosumer processes which is responsible for ingest message and build base cuboid in memory. Could you please show me some detail about how replica Set provide HA guarantee? How to configure it? A link / paper is OK. I found one but I don’t know if it same meaning for your replica Set. a) [Mongodb replication](https://docs.mongodb.com/manual/replication/). 3. How to add or remove node of replica Set in production env? How to monitor the health/pressure of replica Set cluster ? 4. Does all measure are supported in ebay’s New Kylin Streaming Solution? What about count distinct(bitmap)? 5. It seems ebay’s New Kylin Streaming Solution use a custom columnar storage, why not use a open source mature columnar storage solution ? Have your ever compare the performance of your custom columnar storage to open source columnar storage solution ? ---------------- Best wishes, Xiaoxiang Yu 发件人: Ma Gang <[email protected]> 答复: "[email protected]" <[email protected]> 日期: 2018年10月30日 星期二 15:24 收件人: "[email protected]" <[email protected]> 主题: [DISCUSS] New Kylin Streaming Solution From eBay Hi all, eBay Kylin team has developed a new Kylin streaming solution, the basic idea is to build a streaming cluster to ingest data from streaming source(Kafka), and provide query for real-time data, the data preparation latency is milliseconds, which means the data is queryable almost when it is ingested, attach is the architecture design doc. We would like to contribute the feature to community, please let us know if you have any concern. Thanks, Gang(Allen) Ma
