A Streaming interface for building/refreshing the cubes will be an important addition to Kylin capabilities. My personal recommendation though is to harden the existing capabilities with Enterprise features: HA, Security, Performance and SQL capabilities, release a 1.0 and then add Streaming post 1.0.
On 2/2/15, 7:15 AM, "Luke Han" <[email protected]> wrote: >Update a little bit in JIRA for this: >https://issues.apache.org/jira/browse/KYLIN-577 > >Also, there are requirements about integration between Drill/Kylin, >SparkSQL/Kylin and so on. But we are not familiar with these frameworks >for >such integration. >Looking forward some contributors from community to help on these, then we >will weave them into above roadmap. > >Please let's know if you have interesting to join us:-) > >Thanks. > > >Best Regards! >--------------------- > >Luke Han > >2015-02-02 10:58 GMT+08:00 Li Yang <[email protected]>: > >> Streaming cubing is the major difference from Druid. SQL interface and >> Hadoop family add to that. >> >> On Wed, Jan 28, 2015 at 9:36 AM, Luke Han <[email protected]> wrote: >> >> > Thanks Hongbin and Seshu. >> > >> > For the Streaming one, there's proposal from eBay Nous team, please >>refer >> > to this wiki and discussion for more detail: >> > >> > >> >>https://github.com/KylinOLAP/Kylin/wiki/%5BProposal%5D-Kylin-Streaming-Cu >>be-Builder >> > >> > Previous discussion >> > https://groups.google.com/forum/#!topic/kylin-olap/xc7EwdHgmBA >> > >> > The majority goal of Streaming is to accelerate cube build process to >> > reduce data time-to-market for end user. >> > >> > Thanks. >> > Luke >> > >> > 2015-01-27 23:00 GMT+08:00 Adunuthula, Seshu <[email protected]>: >> > >> > > The Roadmap looks great. Some feedback. >> > > >> > > 0.73 Release >> > > - Publish TPC-DS Results. >> > > >> > > 0.8x Release >> > > - Streaming OLAP proposed here, is it different than other offerings >> like >> > > Druid? >> > > - How would Kylin differentiate itself from Druid? >> > > >> > > Regards >> > > Seshu >> > > >> > > On 1/27/15, 6:36 AM, "Luke Han" <[email protected]> wrote: >> > > >> > > >Hi all, >> > > > I have updated a new version of this plan at: >> > > >https://issues.apache.org/jira/browse/KYLIN-577 >> > > > Changes highlights: >> > > > 1. We both agreed to keep v1.0 as more high bar to achieve, >> continue >> > > >to >> > > >use 0.6/0.7/0.8/0.9 for different release target >> > > > 2. Bring DR/Security and others into plan >> > > > 3. Kylin JIRA Project updated with these versions: >> > > > >> > > >> > >> >>https://issues.apache.org/jira/browse/KYLIN/?selectedTab=com.atlassian.ji >>r >> > > >a.jira-projects-plugin:versions-panel >> > > > >> > > > Please kindly help to review and let's know if there's any >>issue. >> > > > >> > > > Thank you very much. >> > > > >> > > >Luke >> > > > >> > > > >> > > >==Release Map== >> > > > >> > > >> > >> >>https://issues.apache.org/jira/secure/attachment/12694778/Kylin%20Release >>% >> > > >20Plan%20-%20H1%202015.png >> > > > >> > > >==Text Copy== >> > > >v0.6.5: MOLAP version >> > > >Goal: >> > > >MOLAP capability to serve SQL query with low latency >> > > >Current version to be released with more stable and bug fixed. So >>far >> > > >there's no critical issues. >> > > >Key features: >> > > >1. Pre-Calculated Cube >> > > >2. Incremental refresh >> > > >3. ANSI SQL Support >> > > >4. Web Interface >> > > >5. ODBC Driver >> > > >Target release date: >> > > >Jan 2015 >> > > > >> > > >v0.7.x: InvertedIndex (HybridOLAP) >> > > >Goal: >> > > >Introduce InvertedIndex to optimise queries on raw data and low >>level >> > > >aggregation >> > > >Support multiple storage and Smart Query Router >> > > >More better user experience >> > > >Key Features: >> > > >1. Bitmap InvertedIndex for raw data and low level aggregation >> > > >2. Logical Model (Data Model) and Physical Model (Storage) >> > > >3. Smart Query Router (Cost-based query router) >> > > >4. New Cube Designer >> > > >5. New Job Manager >> > > >6. JDBC Driver >> > > >Target release date >> > > >Q1 2015 >> > > >v0.7.3: Capacity and Automation >> > > >Goal: >> > > >Capacity management, optimisation and automation >> > > >Key Features: >> > > >1. Quota management >> > > >2. Storage Cleanup and GC >> > > >3. Sampling and Estimation >> > > >4. Cube optimization suggestion >> > > >5. Automate housekeeping jobs >> > > >Target release date >> > > >Q2 2015 >> > > > >> > > >v0.7.5: DR and Security >> > > >Goal: >> > > >Support DR and enhance security >> > > >Key Features: >> > > >1. DR support with DR Cluster >> > > >2. Security enhancement >> > > >Target release date >> > > >TBD >> > > > >> > > >v0.8.x: Streaming and Real-Time (StreamingOLAP) >> > > >Goal: >> > > >Streaming cube building process >> > > >Hybrid Historical and Real Time analysis >> > > >Key Features: >> > > >1. Streaming Cube Builder >> > > >2. Hybrid Query Router >> > > >3. Streaming REST API >> > > >Target release date >> > > >Q2 2015 >> > > > >> > > >v0.9.x: Spark Support >> > > >Goal: >> > > >Leverage Spark tech stack like SparkSQL, Spark Cube Builder >> > > >Key Features: >> > > >1. Spark Cube Builder >> > > >2. TBD >> > > >Target release date >> > > >TBD >> > > > >> > > > >> > > >v1.0: >> > > >Goal: >> > > >Release V1.0 based on community feedback from above release. >> > > >Key Features: >> > > >1. Solid APIs >> > > >2. Solid metadata structure >> > > >3. Easy of use and development >> > > >Target release date >> > > >TBD >> > > > >> > > > >> > > > >> > > > >> > > > >> > > >2015-01-13 22:11 GMT+08:00 Luke Han <[email protected]>: >> > > > >> > > >> I already post in Google Group to encourage people subscript this >> > > >>mailing >> > > >> list. >> > > >> Once more and more people coming to ask about user relative >> questions, >> > > >>it >> > > >> will be good time to create user@ list, i think. >> > > >> >> > > >> Thanks. >> > > >> >> > > >> 2015-01-13 7:04 GMT+08:00 Henry Saputra >><[email protected]>: >> > > >> >> > > >>> I really hope that people whom would like to participate in the >>dev >> > of >> > > >>> Kylin should subscribe to dev@ list. >> > > >>> >> > > >>> Other than that, if a lot of traffic happening about how to use >>or >> > > >>> trouble shoot Kylin, maybe we could start adding user@ list. >> > > >>> >> > > >>> - Henry >> > > >>> >> > > >>> On Mon, Jan 12, 2015 at 2:10 PM, Adunuthula, Seshu >> > > >>><[email protected]> >> > > >>> wrote: >> > > >>> >> I hope Ted's suggestion did not fall to deaf ears. We do not >> need >> > > >>>to cc >> > > >>> >>or >> > > >>> >> plus anybody for conversations in dev@ list. Julian is >>already >> > > >>> subscribed >> > > >>> >> to this list. >> > > >>> > >> > > >>> > >> > > >>> > Understood! >> > > >>> > >> > > >>> > What are your thoughts on including folks into discussions >>that >> do >> > > >>>not >> > > >>> > fall under >> > > >>> > The umbrella of dev >> > > >>> > >> > > >>> >> And diversity should involve individuals not just from MapR, >> > > >>>Cloudera, >> > > >>> or >> > > >>> >> HW. >> > > >>> > >> > > >>> > Couldn’t agree more with this. I know Luke is actively >>working on >> > > >>> this... >> > > >>> > >> > > >>> > >> > > >>> > >> > > >>> > Regards >> > > >>> > Seshu >> > > >>> > >> > > >>> > On 1/12/15, 1:34 PM, "Henry Saputra" <[email protected]> >> > > wrote: >> > > >>> > >> > > >>> >>I hope Ted's suggestion did not fall to deaf ears. We do not >>need >> > to >> > > >>>cc >> > > >>> or >> > > >>> >>plus anybody for conversations in dev@ list. Julian is already >> > > >>> subscribed >> > > >>> >>to this list. >> > > >>> >> >> > > >>> >>And diversity should involve individuals not just from MapR, >> > > >>>Cloudera, >> > > >>> or >> > > >>> >>HW. >> > > >>> >> >> > > >>> >>- Henry >> > > >>> >> >> > > >>> >>On Monday, January 12, 2015, Manjunatha Rao < >> [email protected] >> > > >> > > >>> wrote: >> > > >>> >> >> > > >>> >>> + Julian >> > > >>> >>> >> > > >>> >>> On 1/12/15, 12:58 PM, "Adunuthula, Seshu" < >> [email protected] >> > > >>> >>> <javascript:;>> wrote: >> > > >>> >>> >> > > >>> >>> >On Diversity goal for Kylin would like to explicitly call >>out >> > > >>>MapR¹s >> > > >>> >>> >Involvement and commitment for Kylin and Drill Integration >>and >> > > >>> >>> >specifically >> > > >>> >>> >Ted¹s contribution on guiding Kylin with the Apache Way. >> > > >>> >>> > >> > > >>> >>> >Hoping to see similar commitment from Hortonworks and >> Cloudera. >> > > >>> >>> > >> > > >>> >>> >On 1/12/15, 8:07 AM, "Adunuthula, Seshu" < >> [email protected] >> > > >>> >>> <javascript:;>> wrote: >> > > >>> >>> > >> > > >>> >>> >>Luke, >> > > >>> >>> >> >> > > >>> >>> >>Some thoughts before we tag Kylin as V1.0. We should spend >> some >> > > >>>time >> > > >>> >>>on >> > > >>> >>> >>the fundamentals of Kylin >> > > >>> >>> >> >> > > >>> >>> >>1. Availability: Build out the DR capability in Kylin. We >> have >> > an >> > > >>> >>>option >> > > >>> >>> >>to DR both the offline Cube >> > > >>> >>> >>Build/Refreshes or only the Cube Store (HBase). The >>cheaper >> > > >>>choice >> > > >>> >>>would >> > > >>> >>> >>be to DR the Cube Store (HBase) >> > > >>> >>> >>By creating a Async Replication Instance of HBase on a >> > different >> > > >>> Colo. >> > > >>> >>> >> >> > > >>> >>> >>2. Security: Currently the Kylin Cube security model >>inherits >> > > >>>from >> > > >>> the >> > > >>> >>> >>underlying Hive metadata. This >> > > >>> >>> >>Needs enhancing if we plan to get data from sources other >> than >> > > >>>Hive. >> > > >>> >>> >>Kylin >> > > >>> >>> >>should have its own security >> > > >>> >>> >>Model independent of Hive and we should consider >>integration >> > with >> > > >>> the >> > > >>> >>> >>security solutions out there >> > > >>> >>> >>Knox/Sentry/Ranger etc. >> > > >>> >>> >> >> > > >>> >>> >>3. Performance: We have internal performance numbers, but >> would >> > > >>>like >> > > >>> >>>them >> > > >>> >>> >>validated with externally >> > > >>> >>> >>Recognizable benchmarks such as TPC-DS. >> > > >>> >>> >> >> > > >>> >>> >>4. Diversity: I would like us to focus on diversity of >> > committers >> > > >>> and >> > > >>> >>> >>deployments as the final criteria >> > > >>> >>> >>For the V1 release. >> > > >>> >>> >> >> > > >>> >>> >>Ron and Manju, >> > > >>> >>> >> >> > > >>> >>> >>Would like commitment from Cloudera and Hortonworks >>towards >> > > >>>moving >> > > >>> the >> > > >>> >>> >>needle on Kylin especially >> > > >>> >>> >>On Fundamentals such as Security and Performance. >> > > >>> >>> >> >> > > >>> >>> >>Regards >> > > >>> >>> >>Seshu Adunuthula >> > > >>> >>> >> >> > > >>> >>> >> >> > > >>> >>> >>PS: How come Apache DL emails are still in the 90s. Some >> > > >>>formatting >> > > >>> in >> > > >>> >>> >>the >> > > >>> >>> >>emails would be nice :-) >> > > >>> >>> >> >> > > >>> >>> >> >> > > >>> >>> >>On 1/11/15, 6:52 AM, "Luke Han" <[email protected] >> > > >>><javascript:;>> >> > > >>> >>> wrote: >> > > >>> >>> >> >> > > >>> >>> >>>After open sourced, there are a lot conversion with >> customers, >> > > >>> >>>partners >> > > >>> >>> >>>and >> > > >>> >>> >>>also received many feedback from external, community and >> > others. >> > > >>> >>>Combine >> > > >>> >>> >>>all information we have and considering our bandwidth so >> far, >> > we >> > > >>> >>>drafted >> > > >>> >>> >>>this release plan as below: >> > > >>> >>> >>> >> > > >>> >>> >>> >> > > >>> >> https://www.dropbox.com/sc/nh90bniy7szyvta/AACuv2O3pO1Fog6RwQFfr6YIa >> > > >>> >>> >>> >> > > >>> >>> >>>Here are summary: >> > > >>> >>> >>> >> > > >>> >>> >>> - v1.0 (MOLAP), current version to be released with >>more >> > > >>>stable >> > > >>> >>>and >> > > >>> >>> >>>bug >> > > >>> >>> >>> fixed. So far there's no majority issue. Target >>release >> > date: >> > > >>> >>> >>>2015-01-15 >> > > >>> >>> >>> - v1.1, release via Apache process, target release >>date: >> > > >>> >>>2015-01-31 >> > > >>> >>> >>> - v1.x, bug fix and enhance, soon be merged to HOLAP >> > > >>> >>> >>> - v2.0 (HOLAP), introduced InvertedIndex, since there >>are >> > > >>> totally >> > > >>> >>> >>> different design and concept, we name this release as >> 2.0, >> > > >>> target >> > > >>> >>> >>>release >> > > >>> >>> >>> date: 2015-03-31 >> > > >>> >>> >>> - v2.3, Capacity management and Automation >> > > >>> >>> >>> - v2.5 (Streaming Cube Builder), introduce Streaming >>Cube >> > > >>> Builder >> > > >>> >>> >>> capability in this release >> > > >>> >>> >>> - v2.x, Spark & SparkSQL support, target release date: >> > > >>> 2015-06-30 >> > > >>> >>> >>> >> > > >>> >>> >>> >> > > >>> >>> >>>Please let's know if there any problem and reply here to >> > > >>>discuss. >> > > >>> >>> >>> >> > > >>> >>> >>>Thank you very much >> > > >>> >>> >>> >> > > >>> >>> >>>Luke >> > > >>> >>> >> >> > > >>> >>> > >> > > >>> >>> >> > > >>> >>> >> > > >>> > >> > > >>> >> > > >> >> > > >> >> > > >> > > >> > >> > >> > -- >> > >> > Best Regards! >> > --------------------- >> > >> > Luke Han >> > >>
