[ https://issues.apache.org/jira/browse/KYLIN-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhengdong updated KYLIN-1985: ----------------------------- Description: we suffered from a strange problem that we got a java.lang.ArrayIndexOutOfBoundsException when build of refresh a cube, exception stack like this : java.lang.IllegalStateException: Failed to load lookup table DIM_TABLE_NAME from snapshot /table_snapshot/dim_table_name/5a78a522-6f85-4650-b47d-6a5f5806b7f7.snapshot at org.apache.kylin.cube.CubeManager.getLookupTable(CubeManager.java:621) at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:61) at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:42) at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:56) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:112) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:112) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:127) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.ArrayIndexOutOfBoundsException: 19 at org.apache.kylin.dict.lookup.LookupStringTable.convertRow(LookupStringTable.java:85) at org.apache.kylin.dict.lookup.LookupStringTable.convertRow(LookupStringTable.java:34) at org.apache.kylin.dict.lookup.LookupTable.initRow(LookupTable.java:76) at org.apache.kylin.dict.lookup.LookupTable.init(LookupTable.java:67) at org.apache.kylin.dict.lookup.LookupStringTable.init(LookupStringTable.java:79) at org.apache.kylin.dict.lookup.LookupTable.<init>(LookupTable.java:55) at org.apache.kylin.dict.lookup.LookupStringTable.<init>(LookupStringTable.java:65) at org.apache.kylin.cube.CubeManager.getLookupTable(CubeManager.java:619) ... 13 more and a simple exception when queried by a lookup table dimension ERROR [http-bio-7070-exec-7] controller.QueryController:209 : Exception when execute sql at org.apache.calcite.avatica.Helper.createException(Helper.java:56) at org.apache.calcite.avatica.Helper.createException(Helper.java:41) at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:143) at org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:186) at org.apache.kylin.rest.service.QueryService.execute(QueryService.java:366) at org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:278) at org.apache.kylin.rest.service.QueryService.query(QueryService.java:121) at org.apache.kylin.rest.service.QueryService$$FastClassByCGLIB$$4957273f.invoke(<generated>) at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:618) at org.apache.kylin.rest.service.QueryService$$EnhancerByCGLIB$$315e2079.query(<generated>) at org.apache.kylin.rest.controller.QueryController.doQueryWithCache(QueryController.java:192) at org.apache.kylin.rest.controller.QueryController.query(QueryController.java:94) Caused by: java.lang.ArrayIndexOutOfBoundsException: 19 at org.apache.kylin.dict.lookup.LookupStringTable.convertRow(LookupStringTable.java:85) at org.apache.kylin.dict.lookup.LookupStringTable.convertRow(LookupStringTable.java:34) at org.apache.kylin.dict.lookup.LookupTable.initRow(LookupTable.java:76) at org.apache.kylin.dict.lookup.LookupTable.init(LookupTable.java:67) at org.apache.kylin.dict.lookup.LookupStringTable.init(LookupStringTable.java:79) at org.apache.kylin.dict.lookup.LookupTable.<init>(LookupTable.java:55) at org.apache.kylin.dict.lookup.LookupStringTable.<init>(LookupStringTable.java:65) at org.apache.kylin.cube.CubeManager.getLookupTable(CubeManager.java:619) Though the exception message, we found that one lookup table had been changed in hive (add colunms) and not been synchronized with kylin. However, the cause of this problem is too subtle and not easily found. As for the implementation of was: we suffered from a strange problem that we got a java.lang.ArrayIndexOutOfBoundsException when build of refresh a cube, exception stack like this : java.lang.IllegalStateException: Failed to load lookup table DIM_TABLE_NAME from snapshot /table_snapshot/dim_table_name/5a78a522-6f85-4650-b47d-6a5f5806b7f7.snapshot at org.apache.kylin.cube.CubeManager.getLookupTable(CubeManager.java:621) at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:61) at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:42) at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:56) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:112) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:112) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:127) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.ArrayIndexOutOfBoundsException: 19 at org.apache.kylin.dict.lookup.LookupStringTable.convertRow(LookupStringTable.java:85) at org.apache.kylin.dict.lookup.LookupStringTable.convertRow(LookupStringTable.java:34) at org.apache.kylin.dict.lookup.LookupTable.initRow(LookupTable.java:76) at org.apache.kylin.dict.lookup.LookupTable.init(LookupTable.java:67) at org.apache.kylin.dict.lookup.LookupStringTable.init(LookupStringTable.java:79) at org.apache.kylin.dict.lookup.LookupTable.<init>(LookupTable.java:55) at org.apache.kylin.dict.lookup.LookupStringTable.<init>(LookupStringTable.java:65) at org.apache.kylin.cube.CubeManager.getLookupTable(CubeManager.java:619) ... 13 more and a simple exception when queried by a lookup table dimension ERROR [http-bio-7070-exec-7] controller.QueryController:209 : Exception when execute sql at org.apache.calcite.avatica.Helper.createException(Helper.java:56) at org.apache.calcite.avatica.Helper.createException(Helper.java:41) at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:143) at org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:186) at org.apache.kylin.rest.service.QueryService.execute(QueryService.java:366) at org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:278) at org.apache.kylin.rest.service.QueryService.query(QueryService.java:121) at org.apache.kylin.rest.service.QueryService$$FastClassByCGLIB$$4957273f.invoke(<generated>) at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:618) at org.apache.kylin.rest.service.QueryService$$EnhancerByCGLIB$$315e2079.query(<generated>) at org.apache.kylin.rest.controller.QueryController.doQueryWithCache(QueryController.java:192) at org.apache.kylin.rest.controller.QueryController.query(QueryController.java:94) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:213) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:126) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:96) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:617) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:578) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789) at javax.servlet.http.HttpServlet.service(HttpServlet.java:650) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(DefaultLoginPageGeneratingFilter.java:91) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:205) at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:266) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.ArrayIndexOutOfBoundsException: 19 at org.apache.kylin.dict.lookup.LookupStringTable.convertRow(LookupStringTable.java:85) at org.apache.kylin.dict.lookup.LookupStringTable.convertRow(LookupStringTable.java:34) at org.apache.kylin.dict.lookup.LookupTable.initRow(LookupTable.java:76) at org.apache.kylin.dict.lookup.LookupTable.init(LookupTable.java:67) at org.apache.kylin.dict.lookup.LookupStringTable.init(LookupStringTable.java:79) at org.apache.kylin.dict.lookup.LookupTable.<init>(LookupTable.java:55) at org.apache.kylin.dict.lookup.LookupStringTable.<init>(LookupStringTable.java:65) at org.apache.kylin.cube.CubeManager.getLookupTable(CubeManager.java:619) at org.apache.kylin.storage.hbase.cube.v1.CubeStorageQuery.translateDerivedInCompare(CubeStorageQuery.java:363) at org.apache.kylin.storage.hbase.cube.v1.CubeStorageQuery.translateDerived(CubeStorageQuery.java:321) at org.apache.kylin.storage.hbase.cube.v1.CubeStorageQuery.translateDerived(CubeStorageQuery.java:328) at org.apache.kylin.storage.hbase.cube.v1.CubeStorageQuery.search(CubeStorageQuery.java:138) at org.apache.kylin.query.enumerator.OLAPEnumerator.queryStorage(OLAPEnumerator.java:120) at org.apache.kylin.query.enumerator.OLAPEnumerator.moveNext(OLAPEnumerator.java:69) at Baz$1$1.moveNext(Unknown Source) at org.apache.calcite.linq4j.EnumerableDefaults.aggregate(EnumerableDefaults.java:112) at org.apache.calcite.linq4j.DefaultEnumerable.aggregate(DefaultEnumerable.java:107) at Baz.bind(Unknown Source) at org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:326) at org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:281) at org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:545) at org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:536) at org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:187) at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:65) at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44) at org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:566) at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:578) at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:571) at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:135) Though the exception message, we found that one lookup table had been changed in hive (add colunms) and not been synchronized with kylin. However, it is too subtle and not easily found > SnapshotTable should only keep the colunms described in tableDesc > ----------------------------------------------------------------- > > Key: KYLIN-1985 > URL: https://issues.apache.org/jira/browse/KYLIN-1985 > Project: Kylin > Issue Type: Bug > Affects Versions: v1.5.3 > Reporter: zhengdong > > we suffered from a strange problem that we got a > java.lang.ArrayIndexOutOfBoundsException when build of refresh a cube, > exception stack like this : > java.lang.IllegalStateException: Failed to load lookup table DIM_TABLE_NAME > from snapshot > /table_snapshot/dim_table_name/5a78a522-6f85-4650-b47d-6a5f5806b7f7.snapshot > at > org.apache.kylin.cube.CubeManager.getLookupTable(CubeManager.java:621) > at > org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:61) > at > org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:42) > at > org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:56) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at > org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:112) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:112) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:127) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.ArrayIndexOutOfBoundsException: 19 > at > org.apache.kylin.dict.lookup.LookupStringTable.convertRow(LookupStringTable.java:85) > at > org.apache.kylin.dict.lookup.LookupStringTable.convertRow(LookupStringTable.java:34) > at org.apache.kylin.dict.lookup.LookupTable.initRow(LookupTable.java:76) > at org.apache.kylin.dict.lookup.LookupTable.init(LookupTable.java:67) > at > org.apache.kylin.dict.lookup.LookupStringTable.init(LookupStringTable.java:79) > at org.apache.kylin.dict.lookup.LookupTable.<init>(LookupTable.java:55) > at > org.apache.kylin.dict.lookup.LookupStringTable.<init>(LookupStringTable.java:65) > at > org.apache.kylin.cube.CubeManager.getLookupTable(CubeManager.java:619) > ... 13 more > and a simple exception when queried by a lookup table dimension > > ERROR [http-bio-7070-exec-7] controller.QueryController:209 : Exception when > execute sql > at org.apache.calcite.avatica.Helper.createException(Helper.java:56) > at org.apache.calcite.avatica.Helper.createException(Helper.java:41) > at > org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:143) > at > org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:186) > at > org.apache.kylin.rest.service.QueryService.execute(QueryService.java:366) > at > org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:278) > at > org.apache.kylin.rest.service.QueryService.query(QueryService.java:121) > at > org.apache.kylin.rest.service.QueryService$$FastClassByCGLIB$$4957273f.invoke(<generated>) > at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) > at > org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:618) > at > org.apache.kylin.rest.service.QueryService$$EnhancerByCGLIB$$315e2079.query(<generated>) > at > org.apache.kylin.rest.controller.QueryController.doQueryWithCache(QueryController.java:192) > at > org.apache.kylin.rest.controller.QueryController.query(QueryController.java:94) > Caused by: java.lang.ArrayIndexOutOfBoundsException: 19 > at > org.apache.kylin.dict.lookup.LookupStringTable.convertRow(LookupStringTable.java:85) > at > org.apache.kylin.dict.lookup.LookupStringTable.convertRow(LookupStringTable.java:34) > at > org.apache.kylin.dict.lookup.LookupTable.initRow(LookupTable.java:76) > at org.apache.kylin.dict.lookup.LookupTable.init(LookupTable.java:67) > at > org.apache.kylin.dict.lookup.LookupStringTable.init(LookupStringTable.java:79) > at > org.apache.kylin.dict.lookup.LookupTable.<init>(LookupTable.java:55) > at > org.apache.kylin.dict.lookup.LookupStringTable.<init>(LookupStringTable.java:65) > at > org.apache.kylin.cube.CubeManager.getLookupTable(CubeManager.java:619) > Though the exception message, we found that one lookup table had been changed > in hive (add colunms) and not been synchronized with kylin. However, the > cause of this problem is too subtle and not easily found. As for the > implementation of -- This message was sent by Atlassian JIRA (v6.3.4#6332)