I knew some commercial version of Kylin could store metadata and cube outside of HBase.
On Tue, Jul 11, 2017 at 5:19 AM, Nirav Patel <npa...@xactlycorp.com> wrote: > "Please also note that Kylin's metadata is persisted to HBase by default, > you need have another implementation for ResourceStore." > > I found almost 600 matches for string "hbase" in modules other then > hbase-storage. Hbase seems tightly integrated into modules where it should > not be. Even if it is used for metadata it should been pluggable and not > directly called from other modules. I can't even run DebugTomcat.java > because it relies on classes like AclHBaseStorage (part of server-base) and > calls methods of Hbase storage directly. I think we should make Hbase > dependency injection little bit more pluggable then what it is now. Or use > embedded database like derby or rdbms like mysql which are not so costly to > deploy like hbase. > > Thanks, > Nirav > > On Mon, Jul 10, 2017 at 11:03 AM, Nirav Patel <npa...@xactlycorp.com> > wrote: > > > No equivalent. Instead they came up with Drill - MaprDB connector for > > performance and flexibility. So like earlier version of Kylin I should > be > > able to make drill calls but still do filtering and aggregation on > cluster. > > Only thing is we need to run separate drill-bit service under Yarn for > > this to work. > > > > Thanks, > > Nirav > > > > On Sun, Jul 9, 2017 at 5:06 PM, ShaoFeng Shi <shaofeng...@apache.org> > > wrote: > > > >> Is there some coprocessor-equivalent mechanism in maprdb? If yes, you > can > >> try to implement the logic into it. Otherwise, the performance might be > >> unacceptable when there is bunch of data in each region. > >> > >> I rememer in very early version, Kylin doesn't have coprocessor, all > data > >> be fetched to Kylin server and then do filtering and aggregation by > >> Calcite. Later coprocessor was added and improved continiously. Now > there > >> is no option to disable it. You have to modify the code if don't want to > >> trigger it. > >> > >> > >> > >> 2017-07-09 1:34 GMT+08:00 Nirav Patel <npa...@xactlycorp.com>: > >> > >> > > >> > Thanks Shao for some pointers. I would like some more advice on how I > >> can > >> > replace maprdb as datasource. > >> > > >> > Currently, you can create cubes on maprdb using hbase-storage module > as > >> > maprdb does support hbase client/admin APIs. Only problem is while > >> querying > >> > it throws an error due to Lack of support of Co-processor. > >> > > >> > My initial goal is to get past coprocessor error and return query > >> result. > >> > Then find a better way to implement that strategy - may be a separate > >> > maprdb-storage module. Then look into how to improve performance > without > >> > coprocessors. Any advice on these approaches ? > >> > > >> > Thanks, > >> > Nirav > >> > > >> > On Sat, Jul 8, 2017 at 7:54 AM, ShaoFeng Shi <shaofeng...@apache.org> > >> > wrote: > >> > > >> >> The core modules like core-cube, core-storage are totally independent > >> of > >> >> HBase; while some others like engine-mr, engine-spark has > dependencies > >> on > >> >> HBase. If you want to replace it, you need implement new cubing > engine > >> as > >> >> well. Please also note that Kylin's metadata is persisted to HBase by > >> >> default, you need have another implementation for ResourceStore. > >> >> > >> >> In a short, the plug-in architecture works (we have verfied that), > >> while > >> >> changing the storage is a complex task which take some time to be > >> function > >> >> complete and performance stable. > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> 2017-07-08 5:19 GMT+08:00 Nirav Patel <npa...@xactlycorp.com>: > >> >> > >> >>> Yes, MapR-DB doesn't support coprocessors. > >> >>> > >> >>> Here's the thing though - based on kylin plugin architecture it > >> >>> shouldn't be a problem ideally. Aggregation as well as other DML/DDL > >> >>> operation on datasources should be done transparently. i.e. using > >> >>> kylin-hbase adapter written with calcite or something. It's upto the > >> writer > >> >>> of those adapters to implement aggregates however they want. i.e. > >> either > >> >>> using coprocessors, or in-memory on application server side or using > >> spark. > >> >>> http://kylin.apache.org/development/plugin_arch.html > >> >>> > >> >>> I think for mapr-db we can leverage mapr-drill with secondary > indexes > >> >>> for faster filtering and drill does parallel aggregation as well. > >> Other > >> >>> option is to use mapr-spark with mapr-db which can does the same. > >> >>> > >> >>> Do you know how tightly hbase is coupled with other modules of kylin > >> >>> source other then hbase-storage. > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> On Thu, Jul 6, 2017 at 6:53 PM, ShaoFeng Shi < > shaofeng...@apache.org> > >> >>> wrote: > >> >>> > >> >>>> Hi Nirav, > >> >>>> > >> >>>> I googled that "HBase coprocessors are not present in MapR DB", is > >> >>>> this true? You know Kylin relies on HBase coprocessor to do > filering > >> and > >> >>>> aggregation in each region local; If coprocessor is not available, > >> the > >> >>>> performance will be a problem. > >> >>>> > >> >>>> 2017-07-07 1:29 GMT+08:00 Nirav Patel <npa...@xactlycorp.com>: > >> >>>> > >> >>>>> Hi, We are a mapr users. You will need to deploy separate Hbase > >> >>>>> Cluster alongside your mapr Cluster or on top of it. You won't be > >> able to > >> >>>>> use Mapr-DB with Kylin. > >> >>>>> I am looking into writing Kyling - MaprDB adapter. > >> >>>>> > >> >>>>> On Tue, Jul 4, 2017 at 8:23 AM, <jun....@nomura.com> wrote: > >> >>>>> > >> >>>>>> V5.2, > >> >>>>>> > >> >>>>>> I’m trying to figure out how to deploy Kylin on a MapR cluster, > >> >>>>>> whether Kylin has to be installed on the same cluster of the MapR > >> cluster > >> >>>>>> > >> >>>>>> > >> >>>>>> > >> >>>>>> *From:* Luke Han [mailto:luke...@gmail.com] > >> >>>>>> *Sent:* Tuesday, July 04, 2017 10:04 PM > >> >>>>>> *To:* user > >> >>>>>> *Subject:* Re: Kylin for MapR > >> >>>>>> > >> >>>>>> > >> >>>>>> > >> >>>>>> Kylin support MapR, which MapR version you are using now? > >> >>>>>> > >> >>>>>> > >> >>>>>> > >> >>>>>> > >> >>>>>> Best Regards! > >> >>>>>> --------------------- > >> >>>>>> > >> >>>>>> Luke Han > >> >>>>>> > >> >>>>>> > >> >>>>>> > >> >>>>>> On Tue, Jul 4, 2017 at 4:19 PM, <jun....@nomura.com> wrote: > >> >>>>>> > >> >>>>>> Hi > >> >>>>>> > >> >>>>>> Does kylin support MapR version of the Hadoop? > >> >>>>>> > >> >>>>>> > >> >>>>>> > >> >>>>>> This e-mail (including any attachments) is private and > >> confidential, > >> >>>>>> may contain proprietary or privileged information and is intended > >> for the > >> >>>>>> named recipient(s) only. Unintended recipients are strictly > >> prohibited from > >> >>>>>> taking action on the basis of information in this e-mail and must > >> contact > >> >>>>>> the sender immediately, delete this e-mail (and all attachments) > >> and > >> >>>>>> destroy any hard copies. Nomura will not accept responsibility or > >> liability > >> >>>>>> for the accuracy or completeness of, or the presence of any virus > >> or > >> >>>>>> disabling code in, this e-mail. If verification is sought please > >> request a > >> >>>>>> hard copy. Any reference to the terms of executed transactions > >> should be > >> >>>>>> treated as preliminary only and subject to formal written > >> confirmation by > >> >>>>>> Nomura. Nomura reserves the right to retain, monitor and > intercept > >> e-mail > >> >>>>>> communications through its networks (subject to and in accordance > >> with > >> >>>>>> applicable laws). No confidentiality or privilege is waived or > >> lost by > >> >>>>>> Nomura by any mistransmission of this e-mail. Any reference to > >> "Nomura" is > >> >>>>>> a reference to any entity in the Nomura Holdings, Inc. group. > >> Please read > >> >>>>>> our Electronic Communications Legal Notice which forms part of > >> this e-mail: > >> >>>>>> http://www.Nomura.com/email_disclaimer.htm > >> >>>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www. > >> Nomura.com_email-5Fdisclaimer.htm&d=DwMFaQ&c=dCBwIlVXJsYZrY6 > >> gpNt0LA&r=nFqGYx5SOTBIei-Hwk8uz_JfhmSu2UlelQfdmGX2k5I& > >> m=s6jR2dvhFFTZIuHhCE8WU8EYNQ7M5u-9SggFAoyn5-M&s=iXZr3lmnuJ4X > >> k2-qORnbYZQALda0jRLZ3kzGy816bco&e=> > >> >>>>>> > >> >>>>>> > >> >>>>>> > >> >>>>>> This e-mail (including any attachments) is private and > >> confidential, > >> >>>>>> may contain proprietary or privileged information and is intended > >> for the > >> >>>>>> named recipient(s) only. Unintended recipients are strictly > >> prohibited from > >> >>>>>> taking action on the basis of information in this e-mail and must > >> contact > >> >>>>>> the sender immediately, delete this e-mail (and all attachments) > >> and > >> >>>>>> destroy any hard copies. Nomura will not accept responsibility or > >> liability > >> >>>>>> for the accuracy or completeness of, or the presence of any virus > >> or > >> >>>>>> disabling code in, this e-mail. If verification is sought please > >> request a > >> >>>>>> hard copy. Any reference to the terms of executed transactions > >> should be > >> >>>>>> treated as preliminary only and subject to formal written > >> confirmation by > >> >>>>>> Nomura. Nomura reserves the right to retain, monitor and > intercept > >> e-mail > >> >>>>>> communications through its networks (subject to and in accordance > >> with > >> >>>>>> applicable laws). No confidentiality or privilege is waived or > >> lost by > >> >>>>>> Nomura by any mistransmission of this e-mail. Any reference to > >> "Nomura" is > >> >>>>>> a reference to any entity in the Nomura Holdings, Inc. group. > >> Please read > >> >>>>>> our Electronic Communications Legal Notice which forms part of > >> this e-mail: > >> >>>>>> http://www.Nomura.com/email_disclaimer.htm > >> >>>>>> > >> >>>>> > >> >>>>> > >> >>>>> > >> >>>>> > >> >>>>> [image: What's New with Xactly] > >> >>>>> <http://www.xactlycorp.com/email-click/> > >> >>>>> > >> >>>>> <https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn] > >> >>>>> <https://www.linkedin.com/company/xactly-corporation> [image: > >> >>>>> Twitter] <https://twitter.com/Xactly> [image: Facebook] > >> >>>>> <https://www.facebook.com/XactlyCorp> [image: YouTube] > >> >>>>> <http://www.youtube.com/xactlycorporation> > >> >>>> > >> >>>> > >> >>>> > >> >>>> > >> >>>> -- > >> >>>> Best regards, > >> >>>> > >> >>>> Shaofeng Shi 史少锋 > >> >>>> > >> >>>> > >> >>> > >> >>> > >> >>> > >> >>> [image: What's New with Xactly] <http://www.xactlycorp.com/ema > >> il-click/> > >> >>> > >> >>> <https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn] > >> >>> <https://www.linkedin.com/company/xactly-corporation> [image: > >> Twitter] > >> >>> <https://twitter.com/Xactly> [image: Facebook] > >> >>> <https://www.facebook.com/XactlyCorp> [image: YouTube] > >> >>> <http://www.youtube.com/xactlycorporation> > >> >>> > >> >> > >> >> > >> >> > >> >> -- > >> >> Best regards, > >> >> > >> >> Shaofeng Shi 史少锋 > >> >> > >> >> > >> > > >> > > >> > > >> > [image: What's New with Xactly] <http://www.xactlycorp.com/ > email-click/ > >> > > >> > > >> > <https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn] > >> > <https://www.linkedin.com/company/xactly-corporation> [image: > Twitter] > >> > <https://twitter.com/Xactly> [image: Facebook] > >> > <https://www.facebook.com/XactlyCorp> [image: YouTube] > >> > <http://www.youtube.com/xactlycorporation> > >> > > >> > >> > >> > >> -- > >> Best regards, > >> > >> Shaofeng Shi 史少锋 > >> > > > > > > -- > > > [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/> > > <https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn] > <https://www.linkedin.com/company/xactly-corporation> [image: Twitter] > <https://twitter.com/Xactly> [image: Facebook] > <https://www.facebook.com/XactlyCorp> [image: YouTube] > <http://www.youtube.com/xactlycorporation> >
