JIRA and sub-tasks are created for this. Welcome to comment there: https://issues.apache.org/jira/browse/KYLIN-3621
ShaoFeng Shi <shaofeng...@apache.org> 于2018年10月8日周一 下午2:45写道： > I agree; the new storage should be Hadoop/HDFS compliant, and also need be > cloud storage (like S3, blob storage) friendly, as more and more users are > running big data analytics in the cloud. > > Luke Han <luke...@gmail.com> 于2018年10月7日周日 下午7:44写道： > >> It makes sense to bring a better storage option for Kylin. >> >> The option should be open and people could have different ways to create >> an >> adaptor for the underlying storage. >> Considering huge adoptions of Kylin today are all run on Hadoop/HDFS, I >> prefer for Parquet or ORC or other HDFS compatible option at this time. It >> will easy for people to upgrade to the next generation and keep >> consistency. >> >> Looking forward to this feature to be rolled out soon. >> >> Thanks. >> >> >> >> Best Regards! >> --------------------- >> >> Luke Han >> >> >> On Wed, Oct 3, 2018 at 2:37 PM Li Yang <liy...@apache.org> wrote: >> >> > Love this discussion. Like to highlight 3 major roles HBase is playing >> > currently, so we don't miss any of them when looking for a replacement. >> > >> > 1) Storage: A high speed big data storage >> > 2) Cache: A distributed storage cache layer (was BlockCache) >> > 3) MPP: A distributed computation framework (was Coprocessor) >> > >> > The "Storage" seems at the central of discussion. Be it Parquet, ORC, >> or a >> > new file format, to me the standard interface is most important. As >> long as >> > we have consensus on the access interface, like MapReduce / Spark >> Dataset, >> > then the rest of debate can be easily resolved by a fair benchmark. >> Also it >> > allows people with different preference to keep their own implementation >> > under the standard interface, and not impacting the rest of Kylin. >> > >> > The "Cache" and the "MPP" were more or less overlooked. I suggest we pay >> > more attentions to them. Apart from Spark and Alluxio, any other >> > alternatives? Actually Druid is a well-rounded choice, as like HBase, it >> > covers all the 3 roles pretty well. >> > >> > In general, I prefer to choose from the state of the art instead of >> > re-inventing. Indeed, Kylin is not a storage project. A new storage >> format >> > is not Kylin's mission. Any storage innovations we come across here >> would >> > be more beneficial if contribute to Parquet or ORC community. >> > >> > Regards >> > Yang >> > >> > >> > >> > On Tue, Oct 2, 2018 at 11:20 AM ShaoFeng Shi <shaofeng...@apache.org> >> > wrote: >> > >> > > Hi Billy, >> > > >> > > Yes, the cloud storage should be considered. The traditional file >> layouts >> > > on HDFS may not work well on cloud storage. Kylin needs to allow >> > extension >> > > here. I will add this to the requirement. >> > > >> > > Billy Liu <billy...@apache.org> 于2018年9月29日周六 下午3:22写道： >> > > >> > > > Hi Shaofeng, >> > > > >> > > > I'd like to add one more character: cloud-native storage support. >> > > > Quite a few users are using S3 on AWS, or Azure Data Lake Storage on >> > > > Azure. If new storage engine could be more cloud friendly, more user >> > > > could get benefits from it. >> > > > >> > > > With Warm regards >> > > > >> > > > Billy Liu >> > > > ShaoFeng Shi <shaofeng...@apache.org> 于2018年9月28日周五 下午2:15写道： >> > > > > >> > > > > Hi Kylin developers. >> > > > > >> > > > > HBase has been Kylin’s storage engine since the first day; Kylin >> on >> > > HBase >> > > > > has been verified as a success which can support low latency & >> high >> > > > > concurrency queries on a very large data scale. Thanks to HBase, >> most >> > > > Kylin >> > > > > users can get on average less than 1-second query response. >> > > > > >> > > > > But we also see some limitations when putting Cubes into HBase; I >> > > shared >> > > > > some of them in the HBaseConf Asia 2018[1] this August. The >> typical >> > > > > limitations include: >> > > > > >> > > > > - Rowkey is the primary index, no secondary index so far; >> > > > > >> > > > > Filtering by row key’s prefix and suffix can get very different >> > > > performance >> > > > > result. So the user needs to do a good design about the row key; >> > > > otherwise, >> > > > > the query would be slow. This is difficult sometimes because the >> user >> > > > might >> > > > > not predict the filtering patterns ahead of cube design. >> > > > > >> > > > > - HBase is a key-value instead of a columnar storage >> > > > > >> > > > > Kylin combines multiple measures (columns) into fewer column >> families >> > > for >> > > > > smaller data size (row key size is remarkable). This causes HBase >> > often >> > > > > needing to read more data than requested. >> > > > > >> > > > > - HBase couldn't run on YARN >> > > > > >> > > > > This makes the deployment and auto-scaling a little complicated, >> > > > especially >> > > > > in the cloud. >> > > > > >> > > > > In one word, HBase is complicated to be Kylin’s storage. The >> > > maintenance, >> > > > > debugging is also hard for normal developers. Now we’re planning >> to >> > > seek >> > > > a >> > > > > simple, light-weighted, read-only storage engine for Kylin. The >> new >> > > > > solution should have the following characteristics: >> > > > > >> > > > > - Columnar layout with compression for efficient I/O; >> > > > > - Index by each column for quick filtering and seeking; >> > > > > - MapReduce / Spark API for parallel processing; >> > > > > - HDFS compliant for scalability and availability; >> > > > > - Mature, stable and extensible; >> > > > > >> > > > > With the plugin architecture[2] introduced in Kylin 1.5, adding >> > > multiple >> > > > > storages to Kylin is possible. Some companies like Kyligence Inc >> and >> > > > > Meituan.com, have developed their customized storage engine for >> Kylin >> > > in >> > > > > their product or platform. In their experience, columnar storage >> is a >> > > > good >> > > > > supplement for the HBase engine. Kaisen Kang from Meituan.com has >> > > shared >> > > > > their KOD (Kylin on Druid) solution[3] in this August’s Kylin >> meetup >> > in >> > > > > Beijing. >> > > > > >> > > > > We plan to do a PoC with Apache Parquet + Apache Spark in the next >> > > phase. >> > > > > Parquet is a standard columnar file format and has been widely >> > > supported >> > > > by >> > > > > many projects like Hive, Impala, Drill, etc. Parquet is adding the >> > page >> > > > > level column index to support fine-grained filtering. Apache >> Spark >> > can >> > > > > provide the parallel computing over Parquet and can be deployed on >> > > > > YARN/Mesos and Kubernetes. With this combination, the data >> > persistence >> > > > and >> > > > > computation are separated, which makes the scaling in/out much >> easier >> > > > than >> > > > > before. Benefiting from Spark's flexibility, we can not only push >> > down >> > > > more >> > > > > computation from Kylin to the Hadoop cluster. Except for Parquet, >> > > Apache >> > > > > ORC is also a candidate. >> > > > > >> > > > > Now I raise this discussion to get your ideas about Kylin’s >> > > > next-generation >> > > > > storage engine. If you have good ideas or any related data, >> welcome >> > > > discuss in >> > > > > the community. >> > > > > >> > > > > Thank you! >> > > > > >> > > > > [1] Apache Kylin on HBase >> > > > > >> > > > >> > > >> > >> https://www.slideshare.net/ShiShaoFeng1/apache-kylin-on-hbase-extreme-olap-engine-for-big-data >> > > > > [2] Apache Kylin Plugin Architecture >> > > > > https://kylin.apache.org/development/plugin_arch.html >> > > > > [3] 基于Druid的Kylin存储引擎实践 >> > > > https://blog.bcmeng.com/post/kylin-on-druid.html-- >> > > > > Best regards, >> > > > > >> > > > > Shaofeng Shi 史少锋 >> > > > >> > > >> > > >> > > -- >> > > Best regards, >> > > >> > > Shaofeng Shi 史少锋 >> > > >> > >> > > > -- > Best regards, > > Shaofeng Shi 史少锋 > > -- Best regards, Shaofeng Shi 史少锋
