Hi ShaoFeng,
Thanks a lot for working on the doc. A quick note, when you do get your
fixes in for the next release, I hope the dev team will publish a build vs
1.1.x HBase as you've done for 1.5.x. Many of us have moved on from the
0.98 line.
As for cube building in streaming context, I came to that conclusion
yesterday after some code study. I noticed bin/kylin.sh accepts a
`streaming` option so I pursued that. I think the shell script and the
class it invokes, StreamingCLI, are expecting different command-line
arguments. I tried invoking StreamingCLI directly with my cube, arguments
of `StreamingCLI cube start -start 1462190400 -end 1462190700 -cube
kafka_demo`. This gave me the following log/stack trace.
Thanks a lot for your help!
-n
2016-05-04 20:29:51,387 DEBUG [main HBaseConnection:185]: HTable
'kylin_metadata' already exists
2016-05-04 20:29:51,422 DEBUG [main CubeManager:810]: Loading Cube from
folder kylin_metadata(key='/cube')@kylin_metadata@hbase
2016-05-04 20:29:51,703 INFO [main CubeDescManager:91]: Initializing
CubeDescManager with config kylin_metadata@hbase
2016-05-04 20:29:51,704 INFO [main CubeDescManager:197]: Reloading Cube
Metadata from folder kylin_metadata(key='/cube_desc')@kylin_metadata@hbase
2016-05-04 20:29:51,761 DEBUG [main MetadataManager:264]: Reloading
SourceTable from folder kylin_metadata(key='/table')@kylin_metadata@hbase
2016-05-04 20:29:51,808 DEBUG [main MetadataManager:273]: Loaded 2
SourceTable(s)
2016-05-04 20:29:51,808 DEBUG [main MetadataManager:219]: Reloading
SourceTable exd info from folder
kylin_metadata(key='/table_exd')@kylin_metadata@hbase
2016-05-04 20:29:51,814 DEBUG [main MetadataManager:228]: Loaded 0
SourceTable EXD(s)
2016-05-04 20:29:51,814 DEBUG [main MetadataManager:365]: Reloading
DataModel from folder kylin_metadata(key='/model_desc')@kylin_metadata@hbase
2016-05-04 20:29:51,852 DEBUG [main MetadataManager:379]: Loaded 1
DataModel(s)
2016-05-04 20:29:51,859 DEBUG [main CubeDescManager:222]: Loaded 1 Cube(s)
2016-05-04 20:29:51,861 DEBUG [main CubeManager:840]: Reloaded new cube:
kafka_demo with reference beingCUBE[name=kafka_demo] having 0 segments:
2016-05-04 20:29:51,861 DEBUG [main CubeManager:816]: Loaded 1 Cube(s)
2016-05-04 20:29:51,863 INFO [main StreamingManager:218]: Reloading
Streaming Metadata from folder
kylin_metadata(key='/streaming')@kylin_metadata@hbase
2016-05-04 20:29:51,874 DEBUG [main StreamingManager:243]: Loaded 1
StreamingConfig(s)
2016-05-04 20:29:51,874 WARN [main StreamingCLI:108]: invalid args:cube
start -start 1462190400 -end 1462190700 -cube kafka_demo
2016-05-04 20:29:51,875 ERROR [main StreamingCLI:97]: error start streaming
java.lang.IllegalArgumentException: Table DEFAULT.HTTP is not a streaming
table.
at
org.apache.kylin.source.kafka.KafkaStreamingInput.getBatchWithTimeWindow(KafkaStreamingInput.java:83)
at
org.apache.kylin.engine.streaming.OneOffStreamingBuilder$1.run(OneOffStreamingBuilder.java:75)
at
org.apache.kylin.engine.streaming.cli.StreamingCLI.startOneOffCubeStreaming(StreamingCLI.java:104)
at
org.apache.kylin.engine.streaming.cli.StreamingCLI.main(StreamingCLI.java:91)
2016-05-04 20:29:51,887 INFO [Thread-0
ConnectionManager$HConnectionImplementation:1677]: Closing zookeeper
sessionid=0x1547c9ab1b10015
2016-05-04 20:29:51,889 INFO [Thread-0 ZooKeeper:684]: Session:
0x1547c9ab1b10015 closed
2016-05-04 20:29:51,889 INFO [main-EventThread ClientCnxn:512]:
EventThread shut down
On Thu, May 5, 2016 at 7:33 AM, ShaoFeng Shi <[email protected]> wrote:
> Hello everyone who are interested on this feature; I'm writing a step by
> step tutorial about building cube from kafka in these days; When preparing
> the demo, I found a couple of issues in current v1.5 release; I plan to
> fix/enhance them in next release. The tutorial doc will be published
> together at that time.
>
> Here I would like to answer the question about this error: seems you're
> trying to trigger a cube build in the normal way (which will use MR engine
> to build it); while for a streaming cube, MR wouldn't work as the source
> data is not in Hive; Today we use a command line (bin/streaming_build.sh
> with couple of parameters) to trigger a micro-batch build process, which
> will read data from kafka and put the cube to HBase on finish.
>
> Anyway, please keep tuned for the hot-fix and a step by step guide, believe
> that will save your time and effort; Thanks for the feedback!
>
>
> 2016-05-05 19:22 GMT+08:00 Vaibhav Taro <[email protected]>:
>
> > Did you figure out what was the issue? I am getting the same error while
> > building a streaming cube.
> >
> > On Wed, May 4, 2016 at 4:32 AM, Nick Dimiduk <[email protected]>
> wrote:
> >
> > > For what it's worth, I get the same error in the above stacktrace. I
> > added
> > > a kafka source, defined my model over that source, and then created a
> > cube
> > > from that. With that cube, I tried doing as with the tutorial sample,
> > > submitting a build action. This produces this error.
> > >
> > > On Tue, May 3, 2016 at 3:59 PM, Nick Dimiduk <[email protected]>
> > wrote:
> > >
> > > > On Thu, Apr 14, 2016 at 3:24 AM, ShaoFeng Shi <
> [email protected]>
> > > > wrote:
> > > >
> > > >> For how to setup streaming cubing, please keep updated with the
> > > tutorial.
> > > >>
> > > >
> > > > Where do i find this tutorial? I don't find it on the project site or
> > > > through google search.
> > > >
> > > > Thanks,
> > > > Nick
> > > >
> > > > 2016-04-14 17:39 GMT+08:00 陈佛林 <[email protected]>:
> > > >>
> > > >> > My table is Streaming table.
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > 016-04-14 17:36:42,068 ERROR [http-bio-7070-exec-1]
> > > >> > controller.BasicController:
> > > >> > java.lang.ArrayIndexOutOfBoundsException: 1
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.kylin.common.util.ImplementationSwitch.get(ImplementationSwitch.java:65)
> > > >> > at
> > > >>
> > org.apache.kylin.source.SourceFactory.tableSource(SourceFactory.java:40)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.kylin.source.SourceFactory.createEngineAdapter(SourceFactory.java:48)
> > > >> > at
> > > >> >
> > > >>
> > >
> org.apache.kylin.engine.mr.MRUtil.getBatchCubingInputSide(MRUtil.java:39)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.kylin.engine.mr.BatchCubingJobBuilder2.<init>(BatchCubingJobBuilder2.java:44)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.kylin.engine.mr.MRBatchCubingEngine2.createBatchCubingJob(MRBatchCubingEngine2.java:29)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:56)
> > > >> > at
> > > >>
> > org.apache.kylin.rest.service.JobService.submitJob(JobService.java:211)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.kylin.rest.service.JobService$$FastClassByCGLIB$$83a44b2a.invoke(<generated>)
> > > >> > at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:64)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:622)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.kylin.rest.service.JobService$$EnhancerByCGLIB$$1fccb9aa.submitJob(<generated>)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.kylin.rest.controller.CubeController.rebuild(CubeController.java:214)
> > > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > >> > at java.lang.reflect.Method.invoke(Method.java:606)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:213)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:126)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:96)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:617)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:578)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.web.servlet.FrameworkServlet.doPut(FrameworkServlet.java:800)
> > > >> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:649)
> > > >> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> > > >> > at
> > > >>
> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(DefaultLoginPageGeneratingFilter.java:91)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> > > >> > at
> > > >>
> > com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:195)
> > > >> > at
> > > >>
> > com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:266)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
> > > >> > at
> > > >> >
> > > >>
> > >
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> > > >> > at java.lang.Thread.run(Thread.java:745)
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> > > >> > at java.lang.Thread.run(Thread.java:745)
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Best regards,
> > > >>
> > > >> Shaofeng Shi
> > > >>
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> > VaibhaV
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi
>
