[jira] [Commented] (KYLIN-1340) Tools to extract all cube/hybrid/project related metadata to facilitate diagnosing/debugging/sharing
[ https://issues.apache.org/jira/browse/KYLIN-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15112072#comment-15112072 ] hongbin ma commented on KYLIN-1340: --- another requirement: retrieve a sample of fact table& lookup table to help trouble shooting > Tools to extract all cube/hybrid/project related metadata to facilitate > diagnosing/debugging/sharing > > > Key: KYLIN-1340 > URL: https://issues.apache.org/jira/browse/KYLIN-1340 > Project: Kylin > Issue Type: Improvement >Reporter: hongbin ma >Assignee: hongbin ma > > list of extracted items for a cube: > cube desc > cube instance > data model > all related table desc, and possibly with ext table info > each segment's dict, snapshot,statistics > cube related jobs and job outputs (optional) > list of extracted items for a hybrid: > hybrid instance > for each cube in the hybrid collect "list of extracted items for a cube" > list of extracted items for a project: > project instance > for each cube in the project collect "list of extracted items for a cube" > for each hybrid in the project collect "list of extracted items for a hybrid" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1280) Convert Cuboid Data to HFile failed when hbase in different HDFS
[ https://issues.apache.org/jira/browse/KYLIN-1280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15112070#comment-15112070 ] liyang commented on KYLIN-1280: --- How this differs from KYLIN-957? > Convert Cuboid Data to HFile failed when hbase in different HDFS > > > Key: KYLIN-1280 > URL: https://issues.apache.org/jira/browse/KYLIN-1280 > Project: Kylin > Issue Type: Bug >Affects Versions: 2.0 >Reporter: fengYu > Attachments: > 0001-transform-path-in-other-HDFS-to-real-name-node-path.patch > > > I deploy kylin-2.0 with hbase which rely on a different HDFS with hadoop > cluster, so I config this property 'kylin.hbase.cluster.fs' = hdfs://A, the > name service is different with 'fs.defaultFS' in hadoop cluster which is > hdfs://B. > In the step 'Convert Cuboid Data to HFile' execute failed, error log is : > java.io.IOException: Failed to run job : Unable to map logical nameservice > URI 'hdfs://A' to a NameNode. Local configuration does not have a failover > proxy provide > r configured. > at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:300) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282) > at > org.apache.kylin.engine.mr.common.AbstractHadoopJob.waitForCompletion(AbstractHadoopJob.java:129) > at > org.apache.kylin.storage.hbase.steps.CubeHFileJob.run(CubeHFileJob.java:93) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at > org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:119) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > I think it is because node manager in hadoop cluster can not recognition > hdfs://A in they config. So, I have to tranform the path > hdfs://A/path/to/hfile to hdfs://namenode_ip:port/path/to/hfile before > execute this step. and it works for me. > Here is my patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1339) N/A Job Tracking URL in Build WebUI
[ https://issues.apache.org/jira/browse/KYLIN-1339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15112069#comment-15112069 ] liyang commented on KYLIN-1339: --- My experience is often a retry can get you the correct URL. > N/A Job Tracking URL in Build WebUI > --- > > Key: KYLIN-1339 > URL: https://issues.apache.org/jira/browse/KYLIN-1339 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Reporter: Yerui Sun >Assignee: Zhong,Jason > Labels: newbie > > By designing, the cuboid building steps on web ui has the url directed to the > job page on yarn. > In fact, the url can't be always retrieved successfully. The url is 'N/A' > sometimes, or added a prefix 'unsafe:' sometimes, or the right link > sometimes. With some debug log, founded that the url was fetched from status > info, which filled by job.getTrackingURL() in HadoopCmdOutput.getInfo() when > check job status in MapReduceExecutable. The job tracking url was 'N/A' at > the first 20 seconds since job started, and became the right link after that. > One possible reason is that the tracking url is valid until AppMaster > started, which started slowly. > There's still one question, the job status checking was executed > periodically, and the status info should filled with job tracking url after > some seconds later, but the web ui link was not updated. > Some more digging is needed for this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1339) N/A Job Tracking URL in Build WebUI
[ https://issues.apache.org/jira/browse/KYLIN-1339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyang updated KYLIN-1339: -- Labels: newbie (was: ) > N/A Job Tracking URL in Build WebUI > --- > > Key: KYLIN-1339 > URL: https://issues.apache.org/jira/browse/KYLIN-1339 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Reporter: Yerui Sun >Assignee: Zhong,Jason > Labels: newbie > > By designing, the cuboid building steps on web ui has the url directed to the > job page on yarn. > In fact, the url can't be always retrieved successfully. The url is 'N/A' > sometimes, or added a prefix 'unsafe:' sometimes, or the right link > sometimes. With some debug log, founded that the url was fetched from status > info, which filled by job.getTrackingURL() in HadoopCmdOutput.getInfo() when > check job status in MapReduceExecutable. The job tracking url was 'N/A' at > the first 20 seconds since job started, and became the right link after that. > One possible reason is that the tracking url is valid until AppMaster > started, which started slowly. > There's still one question, the job status checking was executed > periodically, and the status info should filled with job tracking url after > some seconds later, but the web ui link was not updated. > Some more digging is needed for this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1339) N/A Job Tracking URL in Build WebUI
[ https://issues.apache.org/jira/browse/KYLIN-1339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyang updated KYLIN-1339: -- Component/s: (was: Web ) Job Engine > N/A Job Tracking URL in Build WebUI > --- > > Key: KYLIN-1339 > URL: https://issues.apache.org/jira/browse/KYLIN-1339 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Reporter: Yerui Sun >Assignee: Zhong,Jason > Labels: newbie > > By designing, the cuboid building steps on web ui has the url directed to the > job page on yarn. > In fact, the url can't be always retrieved successfully. The url is 'N/A' > sometimes, or added a prefix 'unsafe:' sometimes, or the right link > sometimes. With some debug log, founded that the url was fetched from status > info, which filled by job.getTrackingURL() in HadoopCmdOutput.getInfo() when > check job status in MapReduceExecutable. The job tracking url was 'N/A' at > the first 20 seconds since job started, and became the right link after that. > One possible reason is that the tracking url is valid until AppMaster > started, which started slowly. > There's still one question, the job status checking was executed > periodically, and the status info should filled with job tracking url after > some seconds later, but the web ui link was not updated. > Some more digging is needed for this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1358) revisit on cube selection within same project
[ https://issues.apache.org/jira/browse/KYLIN-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15112066#comment-15112066 ] liangmeng commented on KYLIN-1358: -- we have a case like this: CubeA : date_id,hour_id,service_type,user,count1,count2 the rowkey sequence is : date_id+hour_id+service_type+user this is ok when i select all the users who use serviceA, but if we want to find all the services that userA used, it's became very slow; so we create another cubeB, erverthing is the same only the rowkey sequence changed: CubeB: date_id+hour_id+user+service_type now the problem comes, if i put cubeB in the same project with cubeA, query scene in cubeA become very slow, so we have to build two project! > revisit on cube selection within same project > - > > Key: KYLIN-1358 > URL: https://issues.apache.org/jira/browse/KYLIN-1358 > Project: Kylin > Issue Type: Improvement >Reporter: hongbin ma >Assignee: hongbin ma > > current cost based selection algorithm is far form perfect. need to collect > requirements from the users and rework on it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1082) Hive dependencies should be add to tmpjars
[ https://issues.apache.org/jira/browse/KYLIN-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-1082: -- Attachment: 0001-For-achieving-automatically-upload-hive-related-jars.patch This patch is for 2.x-staging > Hive dependencies should be add to tmpjars > -- > > Key: KYLIN-1082 > URL: https://issues.apache.org/jira/browse/KYLIN-1082 > Project: Kylin > Issue Type: Bug >Reporter: liyang >Assignee: Zhong Yanghong > Labels: newbie > Attachments: > 0001-For-achieving-automatically-upload-hive-related-jars.patch > > > Currently kylin assume all data nodes have hive deployment at exact same FS > location. However, a better position is to think hive as a client side app. > Then we need to ship hive jar with MR job every time. > This make deploy kylin a lot easier in cluster that does not have hive on all > data nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-1358) revisit on cube selection within same project
hongbin ma created KYLIN-1358: - Summary: revisit on cube selection within same project Key: KYLIN-1358 URL: https://issues.apache.org/jira/browse/KYLIN-1358 Project: Kylin Issue Type: Improvement Reporter: hongbin ma Assignee: hongbin ma current cost based selection algorithm is far form perfect. need to collect requirements from the users and rework on it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1357) Cloned cube has build time information
[ https://issues.apache.org/jira/browse/KYLIN-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lola Liu updated KYLIN-1357: Attachment: (was: cube build info mixed up.png) > Cloned cube has build time information > -- > > Key: KYLIN-1357 > URL: https://issues.apache.org/jira/browse/KYLIN-1357 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0 >Reporter: Lola Liu >Assignee: Zhong,Jason > Fix For: v2.0 > > Attachments: cube build info displayed for cloned cube.png > > > STEP: > 1. Login > 2. Clone cube A, name the cloned cube B > 3. Check cube information on model page > RESULT: > The build information is displayed for cube B > (Please refer to attached image) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1357) Cloned cube has build time information
[ https://issues.apache.org/jira/browse/KYLIN-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lola Liu updated KYLIN-1357: Attachment: cube build info displayed for cloned cube.png > Cloned cube has build time information > -- > > Key: KYLIN-1357 > URL: https://issues.apache.org/jira/browse/KYLIN-1357 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0 >Reporter: Lola Liu >Assignee: Zhong,Jason > Fix For: v2.0 > > Attachments: cube build info displayed for cloned cube.png > > > STEP: > 1. Login > 2. Clone cube A, name the cloned cube B > 3. Check cube information on model page > RESULT: > The build information is displayed for cube B > (Please refer to attached image) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1357) Cloned cube has build time information
[ https://issues.apache.org/jira/browse/KYLIN-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lola Liu updated KYLIN-1357: Description: STEP: 1. Login 2. Clone cube A, name the cloned cube B 3. Check cube information on model page RESULT: The build information is displayed for cube B (Please refer to attached image) was: STEP: 1. Login 2. Clone cube A, name the cloned cube B 3. Build cube A 4. Check cube information on model page RESULT: The build information of cube A is displayed for cube B (Please refer to attached image) > Cloned cube has build time information > -- > > Key: KYLIN-1357 > URL: https://issues.apache.org/jira/browse/KYLIN-1357 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0 >Reporter: Lola Liu >Assignee: Zhong,Jason > Fix For: v2.0 > > Attachments: cube build info mixed up.png > > > STEP: > 1. Login > 2. Clone cube A, name the cloned cube B > 3. Check cube information on model page > RESULT: > The build information is displayed for cube B > (Please refer to attached image) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1357) Cloned cube has build time information
[ https://issues.apache.org/jira/browse/KYLIN-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lola Liu updated KYLIN-1357: Summary: Cloned cube has build time information (was: Build information mixed up for cloned cube) > Cloned cube has build time information > -- > > Key: KYLIN-1357 > URL: https://issues.apache.org/jira/browse/KYLIN-1357 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0 >Reporter: Lola Liu >Assignee: Zhong,Jason > Fix For: v2.0 > > Attachments: cube build info mixed up.png > > > STEP: > 1. Login > 2. Clone cube A, name the cloned cube B > 3. Build cube A > 4. Check cube information on model page > RESULT: > The build information of cube A is displayed for cube B > (Please refer to attached image) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1357) Build information mixed up for cloned cube
[ https://issues.apache.org/jira/browse/KYLIN-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lola Liu updated KYLIN-1357: Attachment: cube build info mixed up.png > Build information mixed up for cloned cube > -- > > Key: KYLIN-1357 > URL: https://issues.apache.org/jira/browse/KYLIN-1357 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0 >Reporter: Lola Liu >Assignee: Zhong,Jason > Fix For: v2.0 > > Attachments: cube build info mixed up.png > > > STEP: > 1. Login > 2. Clone cube A, name the cloned cube B > 3. Build cube A > 4. Check cube information on model page > RESULT: > The build information of cube A is displayed for cube B > (Please refer to attached image) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-1357) Build information mixed up for cloned cube
Lola Liu created KYLIN-1357: --- Summary: Build information mixed up for cloned cube Key: KYLIN-1357 URL: https://issues.apache.org/jira/browse/KYLIN-1357 Project: Kylin Issue Type: Bug Components: Web Affects Versions: v2.0 Reporter: Lola Liu Assignee: Zhong,Jason Fix For: v2.0 STEP: 1. Login 2. Clone cube A, name the cloned cube B 3. Build cube A 4. Check cube information on model page RESULT: The build information of cube A is displayed for cube B (Please refer to attached image) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1234) Cube ACL does not work
[ https://issues.apache.org/jira/browse/KYLIN-1234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-1234: Fix Version/s: (was: v2.1) > Cube ACL does not work > -- > > Key: KYLIN-1234 > URL: https://issues.apache.org/jira/browse/KYLIN-1234 > Project: Kylin > Issue Type: Improvement > Components: REST Service >Affects Versions: v1.2, v1.1, v1.1.1 >Reporter: Marek Wiewiorka >Assignee: Shaofeng SHI > Fix For: v2.0, v1.3 > > > Steps to reproduce the issue: > 1)Create a user in the LDAP not assigning it to any group > 2)User is not able to log in the Web UI - since it's not a member of any > default group, nor a member of admin group. > 3)User can however query any cube using REST api. > == > I would expect that unless a user or group has CUBE_QUERY priv granted it > should not be allowed to view the data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KYLIN-1014) Support kerberos authentication while getting status from RM
[ https://issues.apache.org/jira/browse/KYLIN-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-1014. - Resolution: Fixed > Support kerberos authentication while getting status from RM > > > Key: KYLIN-1014 > URL: https://issues.apache.org/jira/browse/KYLIN-1014 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v1.0, v0.7.2, v0.7.1 >Reporter: fengYu >Assignee: Shaofeng SHI > Fix For: v2.0, v1.3 > > Attachments: > 0001-hadoop-status-checker-support-rm-with-kerberos.patch, > patch-for-2.0-rc.patch > > > I have used kylin-0.7.2 build cube and do some query, and I am trying > kylin-1.0 in another hadoop cluster. I get this problem below in kylin-0.7.2 > and kylin-1.0 : > Our hadoop cluster deals with authentication with kerberos, However, We find > after submit a mapreduce job(the second step in building cube), kylin will > send a http request to RM server and get the job status at regular intervals, > But we always get errors here because kylin do nothing about kerberos. > Finally , we do some change on source code and make it support kerberos > authentication . attachment is my patch file.. > I add a property named "kylin.job.status.with.kerberos" which means if we > need do authentication with kerberos when getting status from RM, the default > value is false. > It will be highly appreciated if you have some good idea or some suggestion. > Thanks... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (KYLIN-1279) protobuf jar file conflict with other
[ https://issues.apache.org/jira/browse/KYLIN-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1279. --- Close this JIRA as the problem doesn't exist now; Feel free to reopen if it occurs again. > protobuf jar file conflict with other > - > > Key: KYLIN-1279 > URL: https://issues.apache.org/jira/browse/KYLIN-1279 > Project: Kylin > Issue Type: Bug >Affects Versions: 2.0 >Reporter: fengYu >Assignee: Shaofeng SHI > > I query a sql in kylin-2.0, and error happen, here is the log : > [http-bio-17070-exec-1]:[2016-01-04 > 13:30:13,488][ERROR][org.apache.kylin.rest.controller.BasicController.handleError(BasicController.java:44)] > - > org.apache.kylin.rest.exception.InternalErrorException: loader constraint > violation: loader (instance of org/apache/catalina/loader/WebappClassLoader) > previously initiated loading for a different type with name > "com/google/protobuf/Message" > at > org.apache.kylin.rest.controller.QueryController.doQueryWithCache(QueryController.java:212) > at > org.apache.kylin.rest.controller.QueryController.query(QueryController.java:95) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:213) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:126) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:96) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:617) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:578) > at > org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80) > at > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923) > at > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852) > at > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882) > at > org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:646) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at > org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > I delete protobuf-java-2.5.0.jar file in WEB-INF/lib and restart, it works > fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1014) Support kerberos authentication while getting status from RM
[ https://issues.apache.org/jira/browse/KYLIN-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-1014: Fix Version/s: (was: v2.1) > Support kerberos authentication while getting status from RM > > > Key: KYLIN-1014 > URL: https://issues.apache.org/jira/browse/KYLIN-1014 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v1.0, v0.7.2, v0.7.1 >Reporter: fengYu >Assignee: Shaofeng SHI > Fix For: v2.0, v1.3 > > Attachments: > 0001-hadoop-status-checker-support-rm-with-kerberos.patch, > patch-for-2.0-rc.patch > > > I have used kylin-0.7.2 build cube and do some query, and I am trying > kylin-1.0 in another hadoop cluster. I get this problem below in kylin-0.7.2 > and kylin-1.0 : > Our hadoop cluster deals with authentication with kerberos, However, We find > after submit a mapreduce job(the second step in building cube), kylin will > send a http request to RM server and get the job status at regular intervals, > But we always get errors here because kylin do nothing about kerberos. > Finally , we do some change on source code and make it support kerberos > authentication . attachment is my patch file.. > I add a property named "kylin.job.status.with.kerberos" which means if we > need do authentication with kerberos when getting status from RM, the default > value is false. > It will be highly appreciated if you have some good idea or some suggestion. > Thanks... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KYLIN-1279) protobuf jar file conflict with other
[ https://issues.apache.org/jira/browse/KYLIN-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-1279. - Resolution: Cannot Reproduce > protobuf jar file conflict with other > - > > Key: KYLIN-1279 > URL: https://issues.apache.org/jira/browse/KYLIN-1279 > Project: Kylin > Issue Type: Bug >Affects Versions: 2.0 >Reporter: fengYu >Assignee: Shaofeng SHI > > I query a sql in kylin-2.0, and error happen, here is the log : > [http-bio-17070-exec-1]:[2016-01-04 > 13:30:13,488][ERROR][org.apache.kylin.rest.controller.BasicController.handleError(BasicController.java:44)] > - > org.apache.kylin.rest.exception.InternalErrorException: loader constraint > violation: loader (instance of org/apache/catalina/loader/WebappClassLoader) > previously initiated loading for a different type with name > "com/google/protobuf/Message" > at > org.apache.kylin.rest.controller.QueryController.doQueryWithCache(QueryController.java:212) > at > org.apache.kylin.rest.controller.QueryController.query(QueryController.java:95) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:213) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:126) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:96) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:617) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:578) > at > org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80) > at > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923) > at > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852) > at > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882) > at > org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:646) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at > org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > I delete protobuf-java-2.5.0.jar file in WEB-INF/lib and restart, it works > fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KYLIN-1306) Null strings are not applied during fast cubing
[ https://issues.apache.org/jira/browse/KYLIN-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-1306. - Resolution: Fixed Fix Version/s: v2.0 Fixed, will check whether the value matches with the "null_string" defined in cube_desc > Null strings are not applied during fast cubing > --- > > Key: KYLIN-1306 > URL: https://issues.apache.org/jira/browse/KYLIN-1306 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.0 >Reporter: Dong Li >Assignee: Shaofeng SHI > Fix For: v2.0 > > > One cube defined Null String="NULL" in descriptor, but those rows are not > applied as NULL during fast cubing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KYLIN-1334) allow truncating string for fixed length dimensions
[ https://issues.apache.org/jira/browse/KYLIN-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-1334. - Resolution: Fixed Fixed in both 2.0-rc branch and 2.x-staging branch. > allow truncating string for fixed length dimensions > --- > > Key: KYLIN-1334 > URL: https://issues.apache.org/jira/browse/KYLIN-1334 > Project: Kylin > Issue Type: Improvement >Affects Versions: v2.0 >Reporter: hongbin ma >Assignee: Shaofeng SHI > Fix For: v2.0 > > > in 2.0 above versions, cube building job will fail because it does not > attempt to truncate strings longer than the length() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1128) Clone Cube Metadata
[ https://issues.apache.org/jira/browse/KYLIN-1128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110948#comment-15110948 ] Zhong,Jason commented on KYLIN-1128: in 2.x will also enable clone DataModelDesc, StreamingConfig,Kafkaconfig > Clone Cube Metadata > --- > > Key: KYLIN-1128 > URL: https://issues.apache.org/jira/browse/KYLIN-1128 > Project: Kylin > Issue Type: New Feature > Components: Web >Reporter: Luke Han >Assignee: Zhong,Jason > Fix For: v2.1, v1.3 > > > When user want to duplicate same cube definition for any reason, they have to > create cube from beginning. > To add "Clone Cube" action in web to just copy cube/data model definition > with new name, but not copy any runtime and data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KYLIN-1082) Hive dependencies should be add to tmpjars
[
https://issues.apache.org/jira/browse/KYLIN-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110855#comment-15110855
]
Zhong Yanghong edited comment on KYLIN-1082 at 1/21/16 4:46 PM:
Thank you for you guys. Finally my initial version is done. I mainly changed
the file
engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java.
The additional function is to filter the system level property called
"kylin.hive.dependency" to get interested hive dependency jar names which will
later be added to the hadoop's special property called "tmpjars" whose related
jars will be uploaded to each datanode running mapreduce work.
To add additional helpful jars, there is another way. In the file
"kylin.properties", we can set a property called "kylin.job.mr.lib.dir". Then
AbstractHadoopJob.java will parse out all of the jars and files under this
self-defined directory including the subdirectory and add them to "tmpjars".
was (Author: yaho):
Thank you for you guys. Finally my initial version is done. I mainly changed
the file
engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java.
The additional function is to filter the system level property called
"kylin.hive.dependency" to get interested hive dependency jar names which will
later be added to the hadoop's special property called "tmpjars" whose related
jars will be uploaded to each datanode running mapreduce work.
To set the value for "kylin.hive.dependency", for each platform there is a way.
For development and testing machines which don't install hive and run the
"DebugTomcat.java" to start KYLIN, just add the
System.setProperty("kylin.hive.dependency","") in "DebugTomcat.java". While
for the sandbox which have installed hive and run shell script kylin.sh to
start KYLIN, the shell script find-hive-dependency.sh run in kylin.sh will
automatically set the property.
To add additional helpful jars, there is another way. In the file
"kylin.properties", we can set a property called "kylin.job.mr.lib.dir". Then
AbstractHadoopJob.java will parse out all of the jars and files under this
self-defined directory including the subdirectory and add them to "tmpjars".
> Hive dependencies should be add to tmpjars
> --
>
> Key: KYLIN-1082
> URL: https://issues.apache.org/jira/browse/KYLIN-1082
> Project: Kylin
> Issue Type: Bug
>Reporter: liyang
>Assignee: Zhong Yanghong
> Labels: newbie
>
> Currently kylin assume all data nodes have hive deployment at exact same FS
> location. However, a better position is to think hive as a client side app.
> Then we need to ship hive jar with MR job every time.
> This make deploy kylin a lot easier in cluster that does not have hive on all
> data nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (KYLIN-1082) Hive dependencies should be add to tmpjars
[
https://issues.apache.org/jira/browse/KYLIN-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110855#comment-15110855
]
Zhong Yanghong edited comment on KYLIN-1082 at 1/21/16 4:25 PM:
Thank you for you guys. Finally my initial version is done. I mainly changed
the file
engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java.
The additional function is to filter the system level property called
"kylin.hive.dependency" to get interested hive dependency jar names which will
later be added to the hadoop's special property called "tmpjars" whose related
jars will be uploaded to each datanode running mapreduce work.
To set the value for "kylin.hive.dependency", for each platform there is a way.
For development and testing machines which don't install hive and run the
"DebugTomcat.java" to start KYLIN, just add the
System.setProperty("kylin.hive.dependency","") in "DebugTomcat.java". While
for the sandbox which have installed hive and run shell script kylin.sh to
start KYLIN, the shell script find-hive-dependency.sh run in kylin.sh will
automatically set the property.
To add additional helpful jars, there is another way. In the file
"kylin.properties", we can set a property called "kylin.job.mr.lib.dir". Then
AbstractHadoopJob.java will parse out all of the jars and files under this
self-defined directory including the subdirectory and add them to "tmpjars".
was (Author: yaho):
Thank you for you guys. Finally my initial version is done. I mainly changed
the file
engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java.
The additional function is to filter the system level property called
"kylin.hive.dependency" to get interested hive dependency jar names which will
later be added to the hadoop's special property called "tmpjars" whose related
jars will be uploaded to each datanode running mapreduce work.
To set the value for "kylin.hive.dependency", for each platform there is a way.
For development and testing machines which don't install hive and run the
"DebugTomcat.java" to start KYLIN, just add the
System.setProperty("kylin.hive.dependency",""). While for the sandbox which
have install hive and run kylin.sh to start KYLIN, the shell script
find-hive-dependency.sh run in kylin.sh will automatically set the property.
To add additional helpful jars, there is another way. In the file
"kylin.properties", we can set a property called "kylin.job.mr.lib.dir". Then
AbstractHadoopJob.java will parse out all of the jars and files under this
self-defined directory including the subdirectory and add them to "tmpjars".
> Hive dependencies should be add to tmpjars
> --
>
> Key: KYLIN-1082
> URL: https://issues.apache.org/jira/browse/KYLIN-1082
> Project: Kylin
> Issue Type: Bug
>Reporter: liyang
>Assignee: Zhong Yanghong
> Labels: newbie
>
> Currently kylin assume all data nodes have hive deployment at exact same FS
> location. However, a better position is to think hive as a client side app.
> Then we need to ship hive jar with MR job every time.
> This make deploy kylin a lot easier in cluster that does not have hive on all
> data nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KYLIN-1082) Hive dependencies should be add to tmpjars
[
https://issues.apache.org/jira/browse/KYLIN-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110855#comment-15110855
]
Zhong Yanghong commented on KYLIN-1082:
---
Thank you for you guys. Finally my initial version is done. I mainly changed
the file
engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java.
The additional function is to filter the system level property called
"kylin.hive.dependency" to get interested hive dependency jar names which will
later be added to the hadoop's special property called "tmpjars" whose related
jars will be uploaded to each datanode running mapreduce work.
To set the value for "kylin.hive.dependency", for each platform there is a way.
For development and testing machines which don't install hive and run the
"DebugTomcat.java" to start KYLIN, just add the
System.setProperty("kylin.hive.dependency",""). While for the sandbox which
have install hive and run kylin.sh to start KYLIN, the shell script
find-hive-dependency.sh run in kylin.sh will automatically set the property.
To add additional helpful jars, there is another way. In the file
"kylin.properties", we can set a property called "kylin.job.mr.lib.dir". Then
AbstractHadoopJob.java will parse out all of the jars and files under this
self-defined directory including the subdirectory and add them to "tmpjars".
> Hive dependencies should be add to tmpjars
> --
>
> Key: KYLIN-1082
> URL: https://issues.apache.org/jira/browse/KYLIN-1082
> Project: Kylin
> Issue Type: Bug
>Reporter: liyang
>Assignee: Zhong Yanghong
> Labels: newbie
>
> Currently kylin assume all data nodes have hive deployment at exact same FS
> location. However, a better position is to think hive as a client side app.
> Then we need to ship hive jar with MR job every time.
> This make deploy kylin a lot easier in cluster that does not have hive on all
> data nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (KYLIN-1356) use exec-maven-plugin for IT environment provision
hongbin ma created KYLIN-1356: - Summary: use exec-maven-plugin for IT environment provision Key: KYLIN-1356 URL: https://issues.apache.org/jira/browse/KYLIN-1356 Project: Kylin Issue Type: Bug Reporter: hongbin ma Assignee: hongbin ma currently we rely on test cases like BuildCubeWithEngineTest for IT environment provision. This is not the right way, trying to leverage exec-maven-plugin for this -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1122) Kylin support detail data query from fact table
[ https://issues.apache.org/jira/browse/KYLIN-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110311#comment-15110311 ] liyang commented on KYLIN-1122: --- Add the how to test document: http://kylin.apache.org/development/howto_test.html In this particular case, we want the Raw Measure be covered in integration test. Consider - examples/test_case_data/localmeta/cube/test_kylin_cube_without_slr_empty.json - examples/test_case_data/localmeta/cube/test_kylin_cube_without_slr_left_join_empty.json Seller can be the candidate for Raw Measure. > Kylin support detail data query from fact table > --- > > Key: KYLIN-1122 > URL: https://issues.apache.org/jira/browse/KYLIN-1122 > Project: Kylin > Issue Type: New Feature > Components: Query Engine >Affects Versions: v1.2 >Reporter: Xiaoyu Wang >Assignee: liyang > Fix For: v2.0, v1.3 > > Attachments: > 0001-KYLIN-1122-Kylin-support-detail-data-query-from-fact(2.x-staging).patch, > 0001-KYLIN-1122-Kylin-support-detail-data-query-from-fact(update-v2-1.x-staging).patch, > > 0001-KYLIN-1122-Kylin-support-detail-data-query-from-fact-new-impl-under-refactoring-2.x-staging.patch > > > Now Kylin does not support query correct detail rows from fact table like: > select column1,column2,column3 from fact_table > The jira KYLIN-1075 add the "SUM" function on the measure column if defined. > But only the column number type is support. > I change some code to support this issue: > Add a "VALUE" measure function : the same value and datatype in the input and > output of this function. > If you want to query detail data from fact table > *require*: > 1.Configure the column which not dimensions to "VALUE" or "SUM" measure.(If > not configure measure function in the column will get NULL value) > 2.The source table must has an unique value column and configure it as > dimension. > If you have the better solution please comment here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1314) Display issue for aggression groups
[ https://issues.apache.org/jira/browse/KYLIN-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lola Liu updated KYLIN-1314: Attachment: (was: aggression groups display issue.png) > Display issue for aggression groups > > > Key: KYLIN-1314 > URL: https://issues.apache.org/jira/browse/KYLIN-1314 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0 >Reporter: Lola Liu >Assignee: Zhong,Jason > Fix For: v2.0 > > Attachments: aggression group column display issue.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1314) Display issue for aggression groups
[ https://issues.apache.org/jira/browse/KYLIN-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lola Liu updated KYLIN-1314: Attachment: aggression group column display issue.png > Display issue for aggression groups > > > Key: KYLIN-1314 > URL: https://issues.apache.org/jira/browse/KYLIN-1314 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0 >Reporter: Lola Liu >Assignee: Zhong,Jason > Fix For: v2.0 > > Attachments: aggression group column display issue.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1354) Couldn't edit a cube if it has no "partition date" set
[ https://issues.apache.org/jira/browse/KYLIN-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-1354: Description: Create a cube which has no partition date column, save. Edit this cube, in the "Refresh setting" step, the "Partition Date Column" wasn't set (this is correct), the "Start Date" is "1969-12-31" (this is confusing), the button "next" is disabled without error/warning message. After select a partition column, or empty the "Start Date" field it becomes enabled, while this incorrect or confusing to user: he shouldn't change anything in this page to continue. was: Create a cube which has no partition date column, save. Edit this cube, in the "Refresh setting" step, the "Partition Date Column" wasn't set (this is correct), the "Start Date" is "1969-12-31" (this is confusing), the button "next" is disabled without error/warning message. After select a partition column it becomes enabled, while this incorrect as this cube isn't partitioned. > Couldn't edit a cube if it has no "partition date" set > -- > > Key: KYLIN-1354 > URL: https://issues.apache.org/jira/browse/KYLIN-1354 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v1.2 >Reporter: Shaofeng SHI >Assignee: Zhong,Jason > > Create a cube which has no partition date column, save. > Edit this cube, in the "Refresh setting" step, the "Partition Date Column" > wasn't set (this is correct), the "Start Date" is "1969-12-31" (this is > confusing), the button "next" is disabled without error/warning message. > After select a partition column, or empty the "Start Date" field it becomes > enabled, while this incorrect or confusing to user: he shouldn't change > anything in this page to continue. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-1355) Add script to build a java-hotfix package, instead of a full package
Shaofeng SHI created KYLIN-1355: --- Summary: Add script to build a java-hotfix package, instead of a full package Key: KYLIN-1355 URL: https://issues.apache.org/jira/browse/KYLIN-1355 Project: Kylin Issue Type: Improvement Components: Environment Reporter: Shaofeng SHI Assignee: hongbin ma Sometimes we need deploy only java code change, but he ./scripts/package.sh will make a full binary package which cost extra time on building the web resources. It would be more efficient if we have a way to package only jar files into a tar ball. Then in the server side, we only replace the jars to take effective. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-1354) Couldn't edit a cube if it has no "partition date" set
Shaofeng SHI created KYLIN-1354: --- Summary: Couldn't edit a cube if it has no "partition date" set Key: KYLIN-1354 URL: https://issues.apache.org/jira/browse/KYLIN-1354 Project: Kylin Issue Type: Bug Components: Web Affects Versions: v1.2 Reporter: Shaofeng SHI Assignee: Zhong,Jason Create a cube which has no partition date column, save. Edit this cube, in the "Refresh setting" step, the "Partition Date Column" wasn't set (this is correct), the "Start Date" is "1969-12-31" (this is confusing), the button "next" is disabled without error/warning message. After select a partition column it becomes enabled, while this incorrect as this cube isn't partitioned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
