I agree with your points.. We should make the Adaptor to make it Hbase or Cassandra for Storage option. I will share my thoughts on this. On another line, Can we use Spark QL instead of Apache Calsite for Query engine? This will give better performance than currently as Spark will be running faster. All these should be get controlled by Properties File – based on User’s choice, Kylin code will follow the properties mentioned in the properties file.
I have 3 Suggestions - To read the input data from Parquet File/ORC/Sequence/Text File & Replace Hive table with HDFS file. File Metadata will be come Table Column/Data type & name of Table will be as File-Name. - Spark QL for Kylin query Engine - Cassandra as Storage layer for Kylin Cube processing. Regards, Manoj From: ShaoFeng Shi [mailto:shaofeng...@apache.org] Sent: Thursday, December 14, 2017 8:31 AM To: user Subject: Re: Different backend storage (columnar) - Kylin Improvement Hello, Both HBase and columnar are good options for Kylin's Cube storage, they all have advantages and disadvantages. Columnar is just a format; with it, we still lack a service layer (like HBase service). Some vendor has implemented this by them own, but that is not open source. When having lots of cubes in HBase, remember to review and optimize the Cube design, to minimize the scan and aggregation in HBase; Besides, monitor HBase metrics to know better about the workloads. Usually, set up a dedicated HBase cluster for Kylin would give you more flexibility to do customization and performance tuning, and also be good for stability. Some users are running the whole stack very well, but we also see some is struggling with the operations. Someone raised other option before, like Cassandra, Kudu; but no follow-up. If you have some good idea on this and can make a proposal, that would be nice. Anyway, such discussion and input are always welcomed. 2017-12-09 20:51 GMT+08:00 Kumar, Manoj H <manoj.h.ku...@jpmorgan.com<mailto:manoj.h.ku...@jpmorgan.com>>: Can we include these two points in upcoming release version - Making Kylin Compatible with Spark QL instead of Apache Calcite & having another Cube Storage apart from Hbase. Currently its supporting only Hbase rather its having support for columnar Regards, Manoj From: prasanna lakshmi [mailtoprasannapadar...@gmail.com<mailto:prasannapadar...@gmail.com>] Sent: Saturday, December 09, 2017 10:30 AM To: user@kylin.apache.org<mailto:user@kylin.apache.org> Subject: Re: Different backend storage (columnar) Please make it available different back end database.I am also facing the same issue in cluster. In my clusters physical servers are getting down because of kylin. Please make kylin as light weight tool. It's performance is too good but it needs so many other resources too.My<https://secureweb.jpmchase.net/readonly/http:/too.My> critical situation is all my kylin servers are getting down frequently,so it's affecting other applications also running in the same servers. Can you please tell me in what conditions kylin make servers to down physically. On Dec 9, 2017 3:13 AM, "Sonny Heer" <sonnyh...@gmail.com<mailto:sonnyh...@gmail.com>> wrote: We are using Kylin 1.6 for a while now. One the problems we continue to run into is having to maintain HBase backend. Typically regionservers go down for different reasons. We prefer to move to a columnar storage backend. I heard there was a version of kylin that replaced HBase? Any updates on this and where we can get a hold of this? Thanks This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited. -- Best regards, Shaofeng Shi 史少锋 This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
