[GitHub] [kylin] shaofengshi commented on a change in pull request #582: Read password from file for Beeline
shaofengshi commented on a change in pull request #582: Read password from file for Beeline URL: https://github.com/apache/kylin/pull/582#discussion_r273348735 ## File path: source-hive/src/main/java/org/apache/kylin/source/hive/BeelineHiveClient.java ## @@ -27,6 +27,7 @@ import java.sql.Statement; import java.util.List; import java.util.Properties; +import java.io.*; Review comment: I think "import *" is not allowed in Kylin's static code analysis. Could you please change to use the detailed class import? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3940) Exception in thread "main" java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase
[ https://issues.apache.org/jira/browse/KYLIN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16813049#comment-16813049 ] Chao Long commented on KYLIN-3940: -- Hi [~praveenece], Kylin's hadoop3 binary is for Hadoop 3.0 + HBase 2.0, which is not match your environment. > Exception in thread "main" java.lang.IllegalArgumentException: Failed to find > metadata store by url: kylin_metadata@hbase > - > > Key: KYLIN-3940 > URL: https://issues.apache.org/jira/browse/KYLIN-3940 > Project: Kylin > Issue Type: Test > Components: Environment >Affects Versions: v2.5.2 > Environment: apache-hive-2.3.4-bin > hbase-1.4.9 > hadoop-3.1.0 > apache-kylin-2.5.2-bin-hadoop3 >Reporter: praveenece >Priority: Critical > > I am getting the below error. This error occurred when use > apache-kylin-2.5.2-bin-hadoop3. if we use apache-kylin-2.5.2-bin-hbase1x > kylin gets started. > apache-hive-2.3.4-bin > hbase-1.4.9 > hadoop-3.1.0 > apache-kylin-2.5.2-bin-hadoop3 > Exception in thread "main" java.lang.IllegalArgumentException: Failed to find > metadata store by url: kylin_metadata@hbase > at > org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:98) > at > org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110) > at > org.apache.kylin.rest.service.AclTableMigrationTool.checkIfNeedMigrate(AclTableMigrationTool.java:98) > at > org.apache.kylin.tool.AclTableMigrationCLI.main(AclTableMigrationCLI.java:41) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:92) > ... 3 more > Caused by: java.lang.NoClassDefFoundError: > org/apache/hadoop/hbase/client/ColumnFamilyDescriptor > at > org.apache.kylin.storage.hbase.HBaseResourceStore.getConnection(HBaseResourceStore.java:86) > at > org.apache.kylin.storage.hbase.HBaseResourceStore.createHTableIfNeeded(HBaseResourceStore.java:112) > at > org.apache.kylin.storage.hbase.HBaseResourceStore.(HBaseResourceStore.java:93) > ... 8 more > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.hbase.client.ColumnFamilyDescriptor > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3938) can't delete job OPTIMIZE CHECKPOINT
[ https://issues.apache.org/jira/browse/KYLIN-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16813040#comment-16813040 ] Chao Long commented on KYLIN-3938: -- I will check it. > can't delete job OPTIMIZE CHECKPOINT > > > Key: KYLIN-3938 > URL: https://issues.apache.org/jira/browse/KYLIN-3938 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.6.1 >Reporter: kaishen >Assignee: Chao Long >Priority: Major > Fix For: v2.6.1 > > > I'm using kylin-2.6.1-cdh57 ，when I use cube planner , it will generate tow > jobs one is : OPTIMIZE CUBE another is : OPTIMIZE CHECKPOINT which depend on > job OPTIMIZE CUBE . > now the first job OPTIMIZE CUBE was failed , and I want to discard the > second job , I got error : > java.lang.ClassCastException: > org.apache.kylin.job.execution.CheckpointExecutable cannot be cast to > org.apache.kylin.engine.mr.CubingJob -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3931) Caused by: java.lang.RuntimeException: Could not create interface org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the hadoop compatibility jar o
[ https://issues.apache.org/jira/browse/KYLIN-3931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] praveenece updated KYLIN-3931: -- Environment: apache-hive-2.3.4-bin hbase-1.4.9 hadoop-3.1.0 apache-kylin-2.5.2-bin-hbase1x was: apache-hive-2.3.4-bin hbase-1.4.9 hadoop-3.1.0 apache-kylin-2.5.2-bin-hadoop3 > Caused by: java.lang.RuntimeException: Could not create interface > org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the > hadoop compatibility jar on the classpath? > -- > > Key: KYLIN-3931 > URL: https://issues.apache.org/jira/browse/KYLIN-3931 > Project: Kylin > Issue Type: Test > Components: Environment >Affects Versions: v2.5.2 > Environment: apache-hive-2.3.4-bin > hbase-1.4.9 > hadoop-3.1.0 > apache-kylin-2.5.2-bin-hbase1x >Reporter: praveenece >Priority: Critical > > I am getting the below error. please find the environment details. > apache-hive-2.3.4-bin > hbase-1.4.9 > hadoop-3.1.0 > apache-kylin-2.5.2-bin-hbase1x.tar.gz > Engine Type: Spark > Kylin 63% completed after that i am getting this below error. > WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID 2, server.com, executor > 1): java.lang.ExceptionInInitializerError > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:305) > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:229) > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:167) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123) > at > org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) > at org.apache.spark.scheduler.Task.run(Task.scala:99) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.RuntimeException: Could not create interface > org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the > hadoop compatibility jar on the classpath? > at > org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:73) > at org.apache.hadoop.hbase.io.MetricsIO.(MetricsIO.java:31) > at org.apache.hadoop.hbase.io.hfile.HFile.(HFile.java:191) > ... 15 more > Caused by: java.util.NoSuchElementException > at > java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:365) > at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) > at java.util.ServiceLoader$1.next(ServiceLoader.java:480) > at > org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59) > ... 17 more > 19/04/01 11:47:55 INFO TaskSetManager: Starting task 0.1 in stage 1.0 (TID 3, > server.com, executor 1, partition 0, NODE_LOCAL, 5842 bytes) > 19/04/01 11:47:55 WARN TaskSetManager: Lost task 0.1 in stage 1.0 (TID 3, > server.com, executor 1): java.lang.NoClassDefFoundError: Could not initialize > class org.apache.hadoop.hbase.io.hfile.HFile -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3931) Caused by: java.lang.RuntimeException: Could not create interface org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the hadoop compatibility jar o
[ https://issues.apache.org/jira/browse/KYLIN-3931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] praveenece updated KYLIN-3931: -- Environment: apache-hive-2.3.4-bin hbase-1.4.9 hadoop-3.1.0 apache-kylin-2.5.2-bin-hadoop3 was: apache-hive-2.3.4-bin hbase-1.4.9 hadoop-3.1.0 apache-kylin-2.5.2-bin-hbase1x.tar.gz > Caused by: java.lang.RuntimeException: Could not create interface > org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the > hadoop compatibility jar on the classpath? > -- > > Key: KYLIN-3931 > URL: https://issues.apache.org/jira/browse/KYLIN-3931 > Project: Kylin > Issue Type: Test > Components: Environment >Affects Versions: v2.5.2 > Environment: apache-hive-2.3.4-bin > hbase-1.4.9 > hadoop-3.1.0 > apache-kylin-2.5.2-bin-hadoop3 >Reporter: praveenece >Priority: Critical > > I am getting the below error. please find the environment details. > apache-hive-2.3.4-bin > hbase-1.4.9 > hadoop-3.1.0 > apache-kylin-2.5.2-bin-hbase1x.tar.gz > Engine Type: Spark > Kylin 63% completed after that i am getting this below error. > WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID 2, server.com, executor > 1): java.lang.ExceptionInInitializerError > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:305) > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:229) > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:167) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123) > at > org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) > at org.apache.spark.scheduler.Task.run(Task.scala:99) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.RuntimeException: Could not create interface > org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the > hadoop compatibility jar on the classpath? > at > org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:73) > at org.apache.hadoop.hbase.io.MetricsIO.(MetricsIO.java:31) > at org.apache.hadoop.hbase.io.hfile.HFile.(HFile.java:191) > ... 15 more > Caused by: java.util.NoSuchElementException > at > java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:365) > at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) > at java.util.ServiceLoader$1.next(ServiceLoader.java:480) > at > org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59) > ... 17 more > 19/04/01 11:47:55 INFO TaskSetManager: Starting task 0.1 in stage 1.0 (TID 3, > server.com, executor 1, partition 0, NODE_LOCAL, 5842 bytes) > 19/04/01 11:47:55 WARN TaskSetManager: Lost task 0.1 in stage 1.0 (TID 3, > server.com, executor 1): java.lang.NoClassDefFoundError: Could not initialize > class org.apache.hadoop.hbase.io.hfile.HFile -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3938) can't delete job OPTIMIZE CHECKPOINT
[ https://issues.apache.org/jira/browse/KYLIN-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long reassigned KYLIN-3938: Assignee: Chao Long > can't delete job OPTIMIZE CHECKPOINT > > > Key: KYLIN-3938 > URL: https://issues.apache.org/jira/browse/KYLIN-3938 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.6.1 >Reporter: kaishen >Assignee: Chao Long >Priority: Major > Fix For: v2.6.1 > > > I'm using kylin-2.6.1-cdh57 ，when I use cube planner , it will generate tow > jobs one is : OPTIMIZE CUBE another is : OPTIMIZE CHECKPOINT which depend on > job OPTIMIZE CUBE . > now the first job OPTIMIZE CUBE was failed , and I want to discard the > second job , I got error : > java.lang.ClassCastException: > org.apache.kylin.job.execution.CheckpointExecutable cannot be cast to > org.apache.kylin.engine.mr.CubingJob -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3940) Exception in thread "main" java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase
praveenece created KYLIN-3940: - Summary: Exception in thread "main" java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase Key: KYLIN-3940 URL: https://issues.apache.org/jira/browse/KYLIN-3940 Project: Kylin Issue Type: Test Components: Environment Affects Versions: v2.5.2 Environment: apache-hive-2.3.4-bin hbase-1.4.9 hadoop-3.1.0 apache-kylin-2.5.2-bin-hadoop3 Reporter: praveenece I am getting the below error. This error occurred when use apache-kylin-2.5.2-bin-hadoop3. if we use apache-kylin-2.5.2-bin-hbase1x kylin gets started. apache-hive-2.3.4-bin hbase-1.4.9 hadoop-3.1.0 apache-kylin-2.5.2-bin-hadoop3 Exception in thread "main" java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:98) at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110) at org.apache.kylin.rest.service.AclTableMigrationTool.checkIfNeedMigrate(AclTableMigrationTool.java:98) at org.apache.kylin.tool.AclTableMigrationCLI.main(AclTableMigrationCLI.java:41) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:92) ... 3 more Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/client/ColumnFamilyDescriptor at org.apache.kylin.storage.hbase.HBaseResourceStore.getConnection(HBaseResourceStore.java:86) at org.apache.kylin.storage.hbase.HBaseResourceStore.createHTableIfNeeded(HBaseResourceStore.java:112) at org.apache.kylin.storage.hbase.HBaseResourceStore.(HBaseResourceStore.java:93) ... 8 more Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.client.ColumnFamilyDescriptor at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] asfgit commented on issue #588: Update document
asfgit commented on issue #588: Update document URL: https://github.com/apache/kylin/pull/588#issuecomment-481112113 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] GinaZhai opened a new pull request #588: Update document
GinaZhai opened a new pull request #588: Update document URL: https://github.com/apache/kylin/pull/588 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] asfgit commented on issue #588: Update document
asfgit commented on issue #588: Update document URL: https://github.com/apache/kylin/pull/588#issuecomment-481112114 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] coveralls commented on issue #587: KYLIN-3934 add config for sqoop config null-string and null-non-string
coveralls commented on issue #587: KYLIN-3934 add config for sqoop config null-string and null-non-string URL: https://github.com/apache/kylin/pull/587#issuecomment-481097377 ## Pull Request Test Coverage Report for [Build 4365](https://coveralls.io/builds/22685927) * **0** of **5** **(0.0%)** changed or added relevant lines in **2** files are covered. * **2** unchanged lines in **1** file lost coverage. * Overall coverage decreased (**-0.001%**) to **27.739%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java](https://coveralls.io/builds/22685927/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2FKylinConfigBase.java#L1011) | 0 | 2 | 0.0% | [source-jdbc/src/main/java/org/apache/kylin/source/jdbc/JdbcHiveInputBase.java](https://coveralls.io/builds/22685927/source?filename=source-jdbc%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fsource%2Fjdbc%2FJdbcHiveInputBase.java#L173) | 0 | 3 | 0.0% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DefaultScheduler.java](https://coveralls.io/builds/22685927/source?filename=core-job%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjob%2Fimpl%2Fthreadpool%2FDefaultScheduler.java#L194) | 2 | 80.23% | | Totals | [](https://coveralls.io/builds/22685927) | | :-- | --: | | Change from base [Build 4364](https://coveralls.io/builds/22685230): | -0.001% | | Covered Lines: | 22611 | | Relevant Lines: | 81512 | --- # 💛 - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3934) sqoop import param '--null-string' result in null value become blank string in hive table
[
https://issues.apache.org/jira/browse/KYLIN-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812995#comment-16812995
]
ASF GitHub Bot commented on KYLIN-3934:
---
freewh commented on pull request #587: KYLIN-3934 add config for sqoop config
null-string and null-non-string
URL: https://github.com/apache/kylin/pull/587
add config for sqoop config null-string and null-non-string
fix build error with adding source version and target version in
scala-maven-plugin
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> sqoop import param '--null-string' result in null value become blank string
> in hive table
> -
>
> Key: KYLIN-3934
> URL: https://issues.apache.org/jira/browse/KYLIN-3934
> Project: Kylin
> Issue Type: Bug
> Components: Others
>Affects Versions: v2.6.0
>Reporter: wanghao
>Priority: Major
> Fix For: v2.6.2
>
>
> when column value from jdbc is null, sqoop will write it into hive table with
> blank string.
> eg
> jdbc:
> A | B
> 1 | 1
> 2 | 2
> a | null
>
> hive table:
> A | B
> 1 | 1
> 2 | 2
> a |
> because of this, when I use count(distinct B), it return 3, not 2, and it can
> lead to other problems
>
>
> {code:java}
> String cmd = String.format(Locale.ROOT,
> "%s/bin/sqoop import" + generateSqoopConfigArgString()
> + "--connect \"%s\" --driver %s --username %s --password %s --query \"%s AND
> \\$CONDITIONS\" "
> + "--target-dir %s/%s --split-by %s --boundary-query \"%s\" --null-string '' "
> + "--fields-terminated-by '%s' --num-mappers %d",
> sqoopHome, connectionUrl, driverClass, jdbcUser, jdbcPass, selectSql,
> jobWorkingDir, hiveTable,
> splitColumn, bquery, filedDelimiter, mapperNum);
> {code}
> the param '–null=string' should be '
> \\N' instead of blank string ''.
> I resolved this problem by replace the param. But it needs be configured in
> kylin.properties
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [kylin] asfgit commented on issue #587: KYLIN-3934 add config for sqoop config null-string and null-non-string
asfgit commented on issue #587: KYLIN-3934 add config for sqoop config null-string and null-non-string URL: https://github.com/apache/kylin/pull/587#issuecomment-481092774 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] asfgit commented on issue #587: KYLIN-3934 add config for sqoop config null-string and null-non-string
asfgit commented on issue #587: KYLIN-3934 add config for sqoop config null-string and null-non-string URL: https://github.com/apache/kylin/pull/587#issuecomment-481092773 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] freewh opened a new pull request #587: KYLIN-3934 add config for sqoop config null-string and null-non-string
freewh opened a new pull request #587: KYLIN-3934 add config for sqoop config null-string and null-non-string URL: https://github.com/apache/kylin/pull/587 add config for sqoop config null-string and null-non-string fix build error with adding source version and target version in scala-maven-plugin This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3939) Open csv files from query exported, the Chinese characters are garbled
[ https://issues.apache.org/jira/browse/KYLIN-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812992#comment-16812992 ] zhao jintao commented on KYLIN-3939: I find that if i open csv file using "notepad++", the Chinese characters are displayed normally. Then I convert this file to "UTF-8-BOM", then save this file. Then I open csv file using excel and the Chinese characters are displayed normally. Then I add BOM identifier to the query result to be output in "QueryController.java"，I can open csv files usring excel and it displays normally. The system work fine after this change. I will fix this bug today. Best regards Zhao Jintao > Open csv files from query exported, the Chinese characters are garbled > > > Key: KYLIN-3939 > URL: https://issues.apache.org/jira/browse/KYLIN-3939 > Project: Kylin > Issue Type: Bug > Components: REST Service >Affects Versions: v2.5.2 >Reporter: zhao jintao >Assignee: zhao jintao >Priority: Minor > > Hi team; > I query sql in kylin portal and then press "Export" button to download the > result; the result is csv files. I open the csv file using excel to get the > query results. > But I find that if the result has Chinese characters, the excel will show > Chinese characters garbled. > Does any one face this problem? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3939) Open csv files from query exported, the Chinese characters are garbled
zhao jintao created KYLIN-3939: -- Summary: Open csv files from query exported, the Chinese characters are garbled Key: KYLIN-3939 URL: https://issues.apache.org/jira/browse/KYLIN-3939 Project: Kylin Issue Type: Bug Components: REST Service Affects Versions: v2.5.2 Reporter: zhao jintao Assignee: zhao jintao Hi team; I query sql in kylin portal and then press "Export" button to download the result; the result is csv files. I open the csv file using excel to get the query results. But I find that if the result has Chinese characters, the excel will show Chinese characters garbled. Does any one face this problem? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] zhaojintaozhao commented on issue #583: KYLIN-3788 modify the time conversion time zone of the kafka streaming access; It solves the problems that the kafka stream time is different f
zhaojintaozhao commented on issue #583: KYLIN-3788 modify the time conversion time zone of the kafka streaming access; It solves the problems that the kafka stream time is different from the real time URL: https://github.com/apache/kylin/pull/583#issuecomment-481089274 In my Kylin system, this pull requests solve the problem of time errors in kafka streaming tasks. I hope my code can be fitted into the kylin trunk brach. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3788) Kylin's internal time zone conversions are various and confusing
[
https://issues.apache.org/jira/browse/KYLIN-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812975#comment-16812975
]
zhao jintao commented on KYLIN-3788:
I fix this bug and pull requests to kylin-master：
[https://github.com/apache/kylin/pull/583]。
In my Kylin system, this pull requests solve the problem of time errors in
kafka streaming tasks. I hope my code can be fitted into the kylin trunk
brach.
> Kylin's internal time zone conversions are various and confusing
> -
>
> Key: KYLIN-3788
> URL: https://issues.apache.org/jira/browse/KYLIN-3788
> Project: Kylin
> Issue Type: Bug
> Components: REST Service, Web
>Affects Versions: v2.5.2
> Environment: Huawei FI
>Reporter: zhao jintao
>Assignee: zhao jintao
>Priority: Major
> Labels: TimeZone
>
> Hi, All team
> I found that Kylin has a variety of time zone conversions inside, which is
> confusing. In some code, the time zone is "GMT" when converting time, and
> some code uses the time zone specified in the configuration file
> ("TimeZone.getTimeZone("GMT")" );
> For Example:
> in CubingJob.java, use TimeZone from configuration file:
> format.setTimeZone(TimeZone.getTimeZone(
> config.getTimeZone())),
> but in TimeUtil.java, use GMT TimeZone:private static TimeZone gmt =
> TimeZone.getTimeZone("GMT")
>
> When inport data from Kafka Streaming Data，kylin generate "Data_Start",
> "Hour_Start" uses the GMT time zone, They are some time zones later than
> the real time. Why not use the time zone specified in the configuration file?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [kylin] codecov-io commented on issue #585: KYLIN-3936 MR/Spark task will still run after the job is stopped
codecov-io commented on issue #585: KYLIN-3936 MR/Spark task will still run after the job is stopped URL: https://github.com/apache/kylin/pull/585#issuecomment-481087044 # [Codecov](https://codecov.io/gh/apache/kylin/pull/585?src=pr&el=h1) Report > Merging [#585](https://codecov.io/gh/apache/kylin/pull/585?src=pr&el=desc) into [master](https://codecov.io/gh/apache/kylin/commit/e9dfaf9a5465ff3f6f3bfa3460ca8b2adb8c6617?src=pr&el=desc) will **increase** coverage by `<.01%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/kylin/pull/585?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master #585 +/- ## + Coverage 25.26% 25.26% +<.01% Complexity 5803 5803 Files 1374 1374 Lines 8150881524 +16 Branches 1142911435 +6 + Hits 2059220598 +6 - Misses5890158913 +12 + Partials 2015 2013 -2 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/585?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../apache/kylin/job/execution/ExecutableManager.java](https://codecov.io/gh/apache/kylin/pull/585/diff?src=pr&el=tree#diff-Y29yZS1qb2Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2pvYi9leGVjdXRpb24vRXhlY3V0YWJsZU1hbmFnZXIuamF2YQ==) | `32.32% <0%> (-1.01%)` | `35 <0> (ø)` | | | [...java/org/apache/kylin/rest/service/JobService.java](https://codecov.io/gh/apache/kylin/pull/585/diff?src=pr&el=tree#diff-c2VydmVyLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3Jlc3Qvc2VydmljZS9Kb2JTZXJ2aWNlLmphdmE=) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...rg/apache/kylin/rest/controller/JobController.java](https://codecov.io/gh/apache/kylin/pull/585/diff?src=pr&el=tree#diff-c2VydmVyLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3Jlc3QvY29udHJvbGxlci9Kb2JDb250cm9sbGVyLmphdmE=) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...rg/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://codecov.io/gh/apache/kylin/pull/585/diff?src=pr&el=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2lubWVtY3ViaW5nL01lbURpc2tTdG9yZS5qYXZh) | `70.21% <0%> (+0.91%)` | `7% <0%> (ø)` | :arrow_down: | | [.../apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://codecov.io/gh/apache/kylin/pull/585/diff?src=pr&el=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2N1Ym9pZC9UcmVlQ3Vib2lkU2NoZWR1bGVyLmphdmE=) | `66.15% <0%> (+2.3%)` | `0% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/585?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/585?src=pr&el=footer). Last update [e9dfaf9...224ac4c](https://codecov.io/gh/apache/kylin/pull/585?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3930) ArrayIndexOutOfBoundsException when building
[
https://issues.apache.org/jira/browse/KYLIN-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812945#comment-16812945
]
Jacky Woo commented on KYLIN-3930:
--
hi， [~Shaofengshi] [~Wayne0101]
"what scenario make you still using non-sharded storage type"
My user gave me a building error cube which I found using non-sharded storage
type.
At last I modify the cube desc json file and set storage type to 2.
> ArrayIndexOutOfBoundsException when building
>
>
> Key: KYLIN-3930
> URL: https://issues.apache.org/jira/browse/KYLIN-3930
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: all
>Reporter: Jacky Woo
>Priority: Major
> Fix For: v2.6.2
>
> Attachments: KYLIN-3930.master.01.patch
>
>
> h2. ArrayIndexOutOfBoundsException when building.
> I hive a cube building error with kylin-2.5.0:
> {code:java}
> 2019-03-31 02:45:18,460 ERROR [main] org.apache.kylin.engine.mr.KylinMapper:
> java.lang.ArrayIndexOutOfBoundsException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKeyInternal(NDCuboidBuilder.java:106)
> at
> org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKey(NDCuboidBuilder.java:71)
> at
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:112)
> at
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:47)
> at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> {code}
> I checked the code of "NDCuboidBuilder.buildKeyInternal" method
> {code:java}
> private void buildKeyInternal(Cuboid parentCuboid, Cuboid childCuboid,
> ByteArray[] splitBuffers, ByteArray newKeyBodyBuf) {
> RowKeyEncoder rowkeyEncoder =
> rowKeyEncoderProvider.getRowkeyEncoder(childCuboid);
> // rowkey columns
> long mask = Long.highestOneBit(parentCuboid.getId());
> long parentCuboidId = parentCuboid.getId();
> long childCuboidId = childCuboid.getId();
> long parentCuboidIdActualLength = (long)Long.SIZE -
> Long.numberOfLeadingZeros(parentCuboid.getId());
> int index = rowKeySplitter.getBodySplitOffset(); // skip shard and
> cuboidId
> int offset = RowConstants.ROWKEY_SHARDID_LEN +
> RowConstants.ROWKEY_CUBOIDID_LEN; // skip shard and cuboidId
> for (int i = 0; i < parentCuboidIdActualLength; i++) {
> if ((mask & parentCuboidId) > 0) {// if the this bit position
> equals
> // 1
> if ((mask & childCuboidId) > 0) {// if the child cuboid has
> this
> // column
> System.arraycopy(splitBuffers[index].array(),
> splitBuffers[index].offset(), newKeyBodyBuf.array(), offset,
> splitBuffers[index].length());
> offset += splitBuffers[index].length();
> }
> index++;
> }
> mask = mask >> 1;
> }
> rowkeyEncoder.fillHeader(newKeyBodyBuf.array());
> }
> {code}
> Found that "offset = SHARDID_LEN + CUBOIDID_LEN" , which is wrong when cube
> is not sharding. In my case my cube's storage type is 0, which means it is
> not sharding.
> So, I set offset according to cube sharding, like below:
> {code:java}
> int offset = rowKeySplitter.getHeaderLength(); // skip shard and cuboidId
> {code}
> After modifying building succeeds in my environment.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3938) can't delete job OPTIMIZE CHECKPOINT
kaishen created KYLIN-3938: -- Summary: can't delete job OPTIMIZE CHECKPOINT Key: KYLIN-3938 URL: https://issues.apache.org/jira/browse/KYLIN-3938 Project: Kylin Issue Type: Bug Components: Job Engine Affects Versions: v2.6.1 Reporter: kaishen Fix For: v2.6.1 I'm using kylin-2.6.1-cdh57 ，when I use cube planner , it will generate tow jobs one is : OPTIMIZE CUBE another is : OPTIMIZE CHECKPOINT which depend on job OPTIMIZE CUBE . now the first job OPTIMIZE CUBE was failed , and I want to discard the second job , I got error : java.lang.ClassCastException: org.apache.kylin.job.execution.CheckpointExecutable cannot be cast to org.apache.kylin.engine.mr.CubingJob -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (KYLIN-3924) Build cube on step 'build Dimension Dictionary ', the contents of non-dimensional columns are also checked and built
[ https://issues.apache.org/jira/browse/KYLIN-3924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-3924: - Comment: was deleted (was: Hi, Kylin only build dictionary for dict-encoding columns and some columns in special measure(precisely count distinct, topN) which need global dictionary.) > Build cube on step 'build Dimension Dictionary ', the contents of > non-dimensional columns are also checked and built > > > Key: KYLIN-3924 > URL: https://issues.apache.org/jira/browse/KYLIN-3924 > Project: Kylin > Issue Type: Bug > Components: Job Engine, Tools, Build and Test >Affects Versions: v2.6.1 >Reporter: wangxiaojing >Priority: Major > Labels: build, patch > Attachments: builderr.log > > > Hi all, > I use version 2.6.1 , when build cube on step 'build Dimension Dictionary' > for lookup tables ,it throws some exceptions,it seems checks the content size > of each field in the table while building Dimension Dictionary, and build all > fields in the table with a dictionary. > Suggestion: > 1. Large content fields may not be dimension columns, in fact, it is not > necessary to check the value size of non-dimension columns，only check > dimension columns. > 2. If the value of some fields in the dimension table is too long, the > exception is not clear, can not be located the problem column quickly. > 3. There is no need to build dictionaries for non-dimensional columns -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3906) ExecutableManager is spelled as ExecutableManger
[ https://issues.apache.org/jira/browse/KYLIN-3906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812914#comment-16812914 ] Yanwen Lin commented on KYLIN-3906: --- Sorry for late replying. Sure, I will regenerate patch following the link ASAP. Thanks! > ExecutableManager is spelled as ExecutableManger > > > Key: KYLIN-3906 > URL: https://issues.apache.org/jira/browse/KYLIN-3906 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v3.0.0 >Reporter: Yanwen Lin >Priority: Trivial > Labels: patch > Fix For: v3.0.0 > > Attachments: KYLIN-3906.patch > > > As titled, please see attachment for patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3608) Move dependency versions to top level pom properties
[ https://issues.apache.org/jira/browse/KYLIN-3608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812520#comment-16812520 ] ASF GitHub Bot commented on KYLIN-3608: --- weiqxu commented on pull request #586: KYLIN-3608 Move dependency versions to top level pom properties URL: https://github.com/apache/kylin/pull/586 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move dependency versions to top level pom properties > > > Key: KYLIN-3608 > URL: https://issues.apache.org/jira/browse/KYLIN-3608 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Assignee: zhoujie >Priority: Minor > > There are some non-top level pom.xml files where dependency version is > referenced directly. > core-common/pom.xml is an example. > We should move all dependency versions to top level pom properties -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] asfgit commented on issue #586: KYLIN-3608 Move dependency versions to top level pom properties
asfgit commented on issue #586: KYLIN-3608 Move dependency versions to top level pom properties URL: https://github.com/apache/kylin/pull/586#issuecomment-480886412 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] asfgit commented on issue #586: KYLIN-3608 Move dependency versions to top level pom properties
asfgit commented on issue #586: KYLIN-3608 Move dependency versions to top level pom properties URL: https://github.com/apache/kylin/pull/586#issuecomment-480886411 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] weiqxu opened a new pull request #586: KYLIN-3608 Move dependency versions to top level pom properties
weiqxu opened a new pull request #586: KYLIN-3608 Move dependency versions to top level pom properties URL: https://github.com/apache/kylin/pull/586 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (KYLIN-3831) 唯独超过62生成cuboid 错误
[ https://issues.apache.org/jira/browse/KYLIN-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-3831. --- > 唯独超过62生成cuboid 错误 > - > > Key: KYLIN-3831 > URL: https://issues.apache.org/jira/browse/KYLIN-3831 > Project: Kylin > Issue Type: Wish > Components: Others >Affects Versions: v2.3.1 >Reporter: zhangwei >Assignee: zhangwei >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3930) ArrayIndexOutOfBoundsException when building
[
https://issues.apache.org/jira/browse/KYLIN-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812384#comment-16812384
]
Shaofeng SHI commented on KYLIN-3930:
-
The non-sharded storage are not support after v1.5 I think, though it didn't
report error. Please keep in old version, or switch to the sharded HBase
storage (storage type = 2).
> ArrayIndexOutOfBoundsException when building
>
>
> Key: KYLIN-3930
> URL: https://issues.apache.org/jira/browse/KYLIN-3930
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: all
>Reporter: Jacky Woo
>Priority: Major
> Fix For: v2.6.2
>
> Attachments: KYLIN-3930.master.01.patch
>
>
> h2. ArrayIndexOutOfBoundsException when building.
> I hive a cube building error with kylin-2.5.0:
> {code:java}
> 2019-03-31 02:45:18,460 ERROR [main] org.apache.kylin.engine.mr.KylinMapper:
> java.lang.ArrayIndexOutOfBoundsException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKeyInternal(NDCuboidBuilder.java:106)
> at
> org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKey(NDCuboidBuilder.java:71)
> at
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:112)
> at
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:47)
> at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> {code}
> I checked the code of "NDCuboidBuilder.buildKeyInternal" method
> {code:java}
> private void buildKeyInternal(Cuboid parentCuboid, Cuboid childCuboid,
> ByteArray[] splitBuffers, ByteArray newKeyBodyBuf) {
> RowKeyEncoder rowkeyEncoder =
> rowKeyEncoderProvider.getRowkeyEncoder(childCuboid);
> // rowkey columns
> long mask = Long.highestOneBit(parentCuboid.getId());
> long parentCuboidId = parentCuboid.getId();
> long childCuboidId = childCuboid.getId();
> long parentCuboidIdActualLength = (long)Long.SIZE -
> Long.numberOfLeadingZeros(parentCuboid.getId());
> int index = rowKeySplitter.getBodySplitOffset(); // skip shard and
> cuboidId
> int offset = RowConstants.ROWKEY_SHARDID_LEN +
> RowConstants.ROWKEY_CUBOIDID_LEN; // skip shard and cuboidId
> for (int i = 0; i < parentCuboidIdActualLength; i++) {
> if ((mask & parentCuboidId) > 0) {// if the this bit position
> equals
> // 1
> if ((mask & childCuboidId) > 0) {// if the child cuboid has
> this
> // column
> System.arraycopy(splitBuffers[index].array(),
> splitBuffers[index].offset(), newKeyBodyBuf.array(), offset,
> splitBuffers[index].length());
> offset += splitBuffers[index].length();
> }
> index++;
> }
> mask = mask >> 1;
> }
> rowkeyEncoder.fillHeader(newKeyBodyBuf.array());
> }
> {code}
> Found that "offset = SHARDID_LEN + CUBOIDID_LEN" , which is wrong when cube
> is not sharding. In my case my cube's storage type is 0, which means it is
> not sharding.
> So, I set offset according to cube sharding, like below:
> {code:java}
> int offset = rowKeySplitter.getHeaderLength(); // skip shard and cuboidId
> {code}
> After modifying building succeeds in my environment.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3934) sqoop import param '--null-string' result in null value become blank string in hive table
[
https://issues.apache.org/jira/browse/KYLIN-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812382#comment-16812382
]
Shaofeng SHI commented on KYLIN-3934:
-
Hao, would you like to raise a PR to Kylin? Thank you!
> sqoop import param '--null-string' result in null value become blank string
> in hive table
> -
>
> Key: KYLIN-3934
> URL: https://issues.apache.org/jira/browse/KYLIN-3934
> Project: Kylin
> Issue Type: Bug
> Components: Others
>Affects Versions: v2.6.0
>Reporter: wanghao
>Priority: Major
> Fix For: v2.6.2
>
>
> when column value from jdbc is null, sqoop will write it into hive table with
> blank string.
> eg
> jdbc:
> A | B
> 1 | 1
> 2 | 2
> a | null
>
> hive table:
> A | B
> 1 | 1
> 2 | 2
> a |
> because of this, when I use count(distinct B), it return 3, not 2, and it can
> lead to other problems
>
>
> {code:java}
> String cmd = String.format(Locale.ROOT,
> "%s/bin/sqoop import" + generateSqoopConfigArgString()
> + "--connect \"%s\" --driver %s --username %s --password %s --query \"%s AND
> \\$CONDITIONS\" "
> + "--target-dir %s/%s --split-by %s --boundary-query \"%s\" --null-string '' "
> + "--fields-terminated-by '%s' --num-mappers %d",
> sqoopHome, connectionUrl, driverClass, jdbcUser, jdbcPass, selectSql,
> jobWorkingDir, hiveTable,
> splitColumn, bquery, filedDelimiter, mapperNum);
> {code}
> the param '–null=string' should be '
> \\N' instead of blank string ''.
> I resolved this problem by replace the param. But it needs be configured in
> kylin.properties
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3933) Currently replica set related operation need refresh current front-end page
[ https://issues.apache.org/jira/browse/KYLIN-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-3933: Fix Version/s: v3.0.0 > Currently replica set related operation need refresh current front-end page > --- > > Key: KYLIN-3933 > URL: https://issues.apache.org/jira/browse/KYLIN-3933 > Project: Kylin > Issue Type: Bug > Components: Real-time Streaming, Web >Reporter: Chao Long >Priority: Minor > Fix For: v3.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3936) MR/Spark task will still run after the job is stopped.
[ https://issues.apache.org/jira/browse/KYLIN-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-3936: Fix Version/s: v2.6.2 > MR/Spark task will still run after the job is stopped. > -- > > Key: KYLIN-3936 > URL: https://issues.apache.org/jira/browse/KYLIN-3936 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Fix For: v2.6.2 > > > The command "pause" only sets status of the job to "stopped" and does not > reset the status of the subtask. > So, In SparkExecutable, we can't get the real status of the running task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3818) After Cube disabled, auto-merge cube job still running
[ https://issues.apache.org/jira/browse/KYLIN-3818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-3818. - Resolution: Fixed > After Cube disabled, auto-merge cube job still running > -- > > Key: KYLIN-3818 > URL: https://issues.apache.org/jira/browse/KYLIN-3818 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.6.0 >Reporter: Na Zhai >Assignee: Na Zhai >Priority: Major > Fix For: v3.0.0-alpha > > > *precondition* > There is a Cube that turns on the auto-merge feature. And it satisfied the > condition of the auto-merge. Then the job of merging segments begins. > After a few minutes, the job of merging segments goes into the error status, > so I discard the job. Then I disable this cube, but a new job of merging > segments begin to run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3843) List kylin instances with their server mode on web
[ https://issues.apache.org/jira/browse/KYLIN-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-3843: Fix Version/s: v3.0.0 > List kylin instances with their server mode on web > -- > > Key: KYLIN-3843 > URL: https://issues.apache.org/jira/browse/KYLIN-3843 > Project: Kylin > Issue Type: New Feature > Components: REST Service, Web >Reporter: nichunen >Assignee: Jiatao Tao >Priority: Major > Fix For: v3.0.0 > > > As Curator-based scheduler is available now, so Kylin can list all nodes with > the same metadata url. > This task should include some rest apis to fetch nodes information on ZK, and > front page on System page to display the nodes information. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3936) MR/Spark task will still run after the job is stopped.
[ https://issues.apache.org/jira/browse/KYLIN-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812370#comment-16812370 ] ASF GitHub Bot commented on KYLIN-3936: --- guangxuCheng commented on pull request #585: KYLIN-3936 MR/Spark task will still run after the job is stopped URL: https://github.com/apache/kylin/pull/585 The command "pause" only sets status of the job to "stopped" and does not reset the status of the subtask.https://github.com/apache/kylin/blob/e9dfaf9a5465ff3f6f3bfa3460ca8b2adb8c6617/server-base/src/main/java/org/apache/kylin/rest/service/JobService.java#L662-L666 So, In SparkExecutable, we can't get the real status of the running task. https://github.com/apache/kylin/blob/e9dfaf9a5465ff3f6f3bfa3460ca8b2adb8c6617/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkExecutable.java#L293-L300 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > MR/Spark task will still run after the job is stopped. > -- > > Key: KYLIN-3936 > URL: https://issues.apache.org/jira/browse/KYLIN-3936 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > > The command "pause" only sets status of the job to "stopped" and does not > reset the status of the subtask. > So, In SparkExecutable, we can't get the real status of the running task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] guangxuCheng opened a new pull request #585: KYLIN-3936 MR/Spark task will still run after the job is stopped
guangxuCheng opened a new pull request #585: KYLIN-3936 MR/Spark task will still run after the job is stopped URL: https://github.com/apache/kylin/pull/585 The command "pause" only sets status of the job to "stopped" and does not reset the status of the subtask.https://github.com/apache/kylin/blob/e9dfaf9a5465ff3f6f3bfa3460ca8b2adb8c6617/server-base/src/main/java/org/apache/kylin/rest/service/JobService.java#L662-L666 So, In SparkExecutable, we can't get the real status of the running task. https://github.com/apache/kylin/blob/e9dfaf9a5465ff3f6f3bfa3460ca8b2adb8c6617/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkExecutable.java#L293-L300 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] asfgit commented on issue #585: KYLIN-3936 MR/Spark task will still run after the job is stopped
asfgit commented on issue #585: KYLIN-3936 MR/Spark task will still run after the job is stopped URL: https://github.com/apache/kylin/pull/585#issuecomment-480803549 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] asfgit commented on issue #585: KYLIN-3936 MR/Spark task will still run after the job is stopped
asfgit commented on issue #585: KYLIN-3936 MR/Spark task will still run after the job is stopped URL: https://github.com/apache/kylin/pull/585#issuecomment-480803551 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (KYLIN-3936) MR/Spark task will still run after the job is stopped.
[ https://issues.apache.org/jira/browse/KYLIN-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated KYLIN-3936: - Description: The command "pause" only sets status of the job to "stopped" and does not reset the status of the subtask. So, In SparkExecutable, we can't get the real status of the running task. > MR/Spark task will still run after the job is stopped. > -- > > Key: KYLIN-3936 > URL: https://issues.apache.org/jira/browse/KYLIN-3936 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > > The command "pause" only sets status of the job to "stopped" and does not > reset the status of the subtask. > So, In SparkExecutable, we can't get the real status of the running task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3937) 从 kafka 流构建 cube ,发生死锁
[ https://issues.apache.org/jira/browse/KYLIN-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 风语者 updated KYLIN-3937: --- Description: 你好，用 kylin 从 kafka 流构建 cube, 每五分钟 构建一次，从日志看到好像发生了死锁现象。 目前为止，发生了再次，第一次我将所有正在进行的构建全部 discard , 然后死锁现象还是存在，目前不知道该怎么处理这个问题。烦请查看下吧。多谢多谢。 用 spark 进行构建，到第三步：Build Dimension Dictionary 时，发生了死锁。 发生死锁时，没有发生 cube 构建错误现象。 was: 你好，用 kylin 从 kafka 流构建 cube, 每五分钟 构建一次，从日志看到好像发生了死锁现象。 目前为止，发生了再次，第一次我将所有正在进行的构建全部 discard , 然后死锁现象还是存在，目前不知道该怎么处理这个问题。烦请查看下吧。多谢多谢。 用 spark 进行构建，到第三步：Build Dimension Dictionary 时，发生了死锁。 > 从 kafka 流构建 cube ,发生死锁 > -- > > Key: KYLIN-3937 > URL: https://issues.apache.org/jira/browse/KYLIN-3937 > Project: Kylin > Issue Type: Bug > Components: Job Engine, Metadata, Spark Engine >Affects Versions: v2.6.0 > Environment: kylin: 2.6.0 > hdp: 2.5.0.0-1245 >Reporter: 风语者 >Priority: Critical > Attachments: deadlock_kylin.png, kylin_monitors.png > > > 你好，用 kylin 从 kafka 流构建 cube, 每五分钟 构建一次，从日志看到好像发生了死锁现象。 > 目前为止，发生了再次，第一次我将所有正在进行的构建全部 discard , 然后死锁现象还是存在，目前不知道该怎么处理这个问题。烦请查看下吧。多谢多谢。 > 用 spark 进行构建，到第三步：Build Dimension Dictionary 时，发生了死锁。 > 发生死锁时，没有发生 cube 构建错误现象。 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3937) 从 kafka 流构建 cube ,发生死锁
[ https://issues.apache.org/jira/browse/KYLIN-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 风语者 updated KYLIN-3937: --- Description: 你好，用 kylin 从 kafka 流构建 cube, 每五分钟 构建一次，从日志看到好像发生了死锁现象。 目前为止，发生了再次，第一次我将所有正在进行的构建全部 discard , 然后死锁现象还是存在，目前不知道该怎么处理这个问题。烦请查看下吧。多谢多谢。 用 spark 进行构建，到第三步：Build Dimension Dictionary 时，发生了死锁。 was: 你好，用 kylin 从 kafka 流构建 cube, 每五分钟 构建一次，从日志看到好像发生了死锁现象。 目前为止，发生了再次，第一次我将所有正在进行的构建全部 discard , 然后死锁现象还是存在，目前不知道该怎么处理这个问题。烦请查看下吧。多谢多谢。 > 从 kafka 流构建 cube ,发生死锁 > -- > > Key: KYLIN-3937 > URL: https://issues.apache.org/jira/browse/KYLIN-3937 > Project: Kylin > Issue Type: Bug > Components: Job Engine, Metadata, Spark Engine >Affects Versions: v2.6.0 > Environment: kylin: 2.6.0 > hdp: 2.5.0.0-1245 >Reporter: 风语者 >Priority: Critical > Attachments: deadlock_kylin.png, kylin_monitors.png > > > 你好，用 kylin 从 kafka 流构建 cube, 每五分钟 构建一次，从日志看到好像发生了死锁现象。 > 目前为止，发生了再次，第一次我将所有正在进行的构建全部 discard , 然后死锁现象还是存在，目前不知道该怎么处理这个问题。烦请查看下吧。多谢多谢。 > 用 spark 进行构建，到第三步：Build Dimension Dictionary 时，发生了死锁。 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3937) 从 kafka 流构建 cube ,发生死锁
风语者 created KYLIN-3937: -- Summary: 从 kafka 流构建 cube ,发生死锁 Key: KYLIN-3937 URL: https://issues.apache.org/jira/browse/KYLIN-3937 Project: Kylin Issue Type: Bug Components: Job Engine, Metadata, Spark Engine Affects Versions: v2.6.0 Environment: kylin: 2.6.0 hdp: 2.5.0.0-1245 Reporter: 风语者 Attachments: deadlock_kylin.png, kylin_monitors.png 你好，用 kylin 从 kafka 流构建 cube, 每五分钟 构建一次，从日志看到好像发生了死锁现象。 目前为止，发生了再次，第一次我将所有正在进行的构建全部 discard , 然后死锁现象还是存在，目前不知道该怎么处理这个问题。烦请查看下吧。多谢多谢。 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3873) Fix inappropriate use of memory in SparkFactDistinct.java
[ https://issues.apache.org/jira/browse/KYLIN-3873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812318#comment-16812318 ] Shaofeng SHI commented on KYLIN-3873: - [~Wayne0101] please make this to 2.6.x branch; the commit couldn't be directly applied in 2.6.x > Fix inappropriate use of memory in SparkFactDistinct.java > -- > > Key: KYLIN-3873 > URL: https://issues.apache.org/jira/browse/KYLIN-3873 > Project: Kylin > Issue Type: Improvement >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v2.6.2 > > > Class SparkFactDistinct.java has some inappropriate use of memory -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3930) ArrayIndexOutOfBoundsException when building
[
https://issues.apache.org/jira/browse/KYLIN-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812316#comment-16812316
]
Chao Long commented on KYLIN-3930:
--
Hi [~jackyWoo], We know, sharded storage type was set as default storage type
since Kylin-1.5.0, And I see the Kylin you are using is v2.5.0. Could you tell
me what scenario make you still using non-sharded storage type.
> ArrayIndexOutOfBoundsException when building
>
>
> Key: KYLIN-3930
> URL: https://issues.apache.org/jira/browse/KYLIN-3930
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: all
>Reporter: Jacky Woo
>Priority: Major
> Fix For: v2.6.2
>
> Attachments: KYLIN-3930.master.01.patch
>
>
> h2. ArrayIndexOutOfBoundsException when building.
> I hive a cube building error with kylin-2.5.0:
> {code:java}
> 2019-03-31 02:45:18,460 ERROR [main] org.apache.kylin.engine.mr.KylinMapper:
> java.lang.ArrayIndexOutOfBoundsException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKeyInternal(NDCuboidBuilder.java:106)
> at
> org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKey(NDCuboidBuilder.java:71)
> at
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:112)
> at
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:47)
> at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> {code}
> I checked the code of "NDCuboidBuilder.buildKeyInternal" method
> {code:java}
> private void buildKeyInternal(Cuboid parentCuboid, Cuboid childCuboid,
> ByteArray[] splitBuffers, ByteArray newKeyBodyBuf) {
> RowKeyEncoder rowkeyEncoder =
> rowKeyEncoderProvider.getRowkeyEncoder(childCuboid);
> // rowkey columns
> long mask = Long.highestOneBit(parentCuboid.getId());
> long parentCuboidId = parentCuboid.getId();
> long childCuboidId = childCuboid.getId();
> long parentCuboidIdActualLength = (long)Long.SIZE -
> Long.numberOfLeadingZeros(parentCuboid.getId());
> int index = rowKeySplitter.getBodySplitOffset(); // skip shard and
> cuboidId
> int offset = RowConstants.ROWKEY_SHARDID_LEN +
> RowConstants.ROWKEY_CUBOIDID_LEN; // skip shard and cuboidId
> for (int i = 0; i < parentCuboidIdActualLength; i++) {
> if ((mask & parentCuboidId) > 0) {// if the this bit position
> equals
> // 1
> if ((mask & childCuboidId) > 0) {// if the child cuboid has
> this
> // column
> System.arraycopy(splitBuffers[index].array(),
> splitBuffers[index].offset(), newKeyBodyBuf.array(), offset,
> splitBuffers[index].length());
> offset += splitBuffers[index].length();
> }
> index++;
> }
> mask = mask >> 1;
> }
> rowkeyEncoder.fillHeader(newKeyBodyBuf.array());
> }
> {code}
> Found that "offset = SHARDID_LEN + CUBOIDID_LEN" , which is wrong when cube
> is not sharding. In my case my cube's storage type is 0, which means it is
> not sharding.
> So, I set offset according to cube sharding, like below:
> {code:java}
> int offset = rowKeySplitter.getHeaderLength(); // skip shard and cuboidId
> {code}
> After modifying building succeeds in my environment.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3841) Build Global Dict by MR/Hive
[ https://issues.apache.org/jira/browse/KYLIN-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812312#comment-16812312 ] ASF subversion and git services commented on KYLIN-3841: Commit e9dfaf9a5465ff3f6f3bfa3460ca8b2adb8c6617 in kylin's branch refs/heads/master from hit-lacus [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=e9dfaf9 ] KYLIN-3841 Enable build global dict by mr in CI > Build Global Dict by MR/Hive > > > Key: KYLIN-3841 > URL: https://issues.apache.org/jira/browse/KYLIN-3841 > Project: Kylin > Issue Type: Improvement >Reporter: jinguowei >Assignee: jinguowei >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3841) Build Global Dict by MR/Hive
[ https://issues.apache.org/jira/browse/KYLIN-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812310#comment-16812310 ] ASF subversion and git services commented on KYLIN-3841: Commit 480d80b672d6c91429e56542f24681edf84f309f in kylin's branch refs/heads/master from javalife0312 [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=480d80b ] KYLIN-3841 Build Global Dict by MR/Hive > Build Global Dict by MR/Hive > > > Key: KYLIN-3841 > URL: https://issues.apache.org/jira/browse/KYLIN-3841 > Project: Kylin > Issue Type: Improvement >Reporter: jinguowei >Assignee: jinguowei >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3841) Build Global Dict by MR/Hive
[ https://issues.apache.org/jira/browse/KYLIN-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812309#comment-16812309 ] ASF GitHub Bot commented on KYLIN-3841: --- nichunen commented on pull request #568: KYLIN-3841 Build Global Dict by Hive URL: https://github.com/apache/kylin/pull/568 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Build Global Dict by MR/Hive > > > Key: KYLIN-3841 > URL: https://issues.apache.org/jira/browse/KYLIN-3841 > Project: Kylin > Issue Type: Improvement >Reporter: jinguowei >Assignee: jinguowei >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3841) Build Global Dict by MR/Hive
[ https://issues.apache.org/jira/browse/KYLIN-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812311#comment-16812311 ] ASF subversion and git services commented on KYLIN-3841: Commit 5d04f2febbfa50346da56e3be7b2211a0b5d8071 in kylin's branch refs/heads/master from hit-lacus [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=5d04f2f ] KYLIN-3841 Build Global Dict by MR/Hive > Build Global Dict by MR/Hive > > > Key: KYLIN-3841 > URL: https://issues.apache.org/jira/browse/KYLIN-3841 > Project: Kylin > Issue Type: Improvement >Reporter: jinguowei >Assignee: jinguowei >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] nichunen merged pull request #568: KYLIN-3841 Build Global Dict by Hive
nichunen merged pull request #568: KYLIN-3841 Build Global Dict by Hive URL: https://github.com/apache/kylin/pull/568 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (KYLIN-3936) MR/Spark task will still run after the job is stopped.
Guangxu Cheng created KYLIN-3936: Summary: MR/Spark task will still run after the job is stopped. Key: KYLIN-3936 URL: https://issues.apache.org/jira/browse/KYLIN-3936 Project: Kylin Issue Type: Bug Components: Job Engine Reporter: Guangxu Cheng Assignee: Guangxu Cheng -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3918) Add project name in cube and job pages
[ https://issues.apache.org/jira/browse/KYLIN-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812248#comment-16812248 ] Liu Shaohui commented on KYLIN-3918: [~Shaofengshi] In my opinion, it not too wide. The project info is useful and I can't not find another more suitable position. Any advice about the position? Thanks~ > Add project name in cube and job pages > -- > > Key: KYLIN-3918 > URL: https://issues.apache.org/jira/browse/KYLIN-3918 > Project: Kylin > Issue Type: Improvement >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Fix For: v2.6.2 > > > In a production cluster, there will be many projects and each project has > many cubes. It's useful to show project name in cube and job pages. > So the admin can be quick to known which project the abnormal cube or failed > job belongs to and get contact with the users. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3935) ZKUtil acquire the wrong Zookeeper Path on windows
[
https://issues.apache.org/jira/browse/KYLIN-3935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812213#comment-16812213
]
ASF subversion and git services commented on KYLIN-3935:
Commit 80ac894f36bd26780564c04a2eae43a7146a50af in kylin's branch
refs/heads/master from GinaZhai
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=80ac894 ]
KYLIN-3935 ZKUtil acquire the wrong Zookeeper Path on windows
> ZKUtil acquire the wrong Zookeeper Path on windows
> --
>
> Key: KYLIN-3935
> URL: https://issues.apache.org/jira/browse/KYLIN-3935
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.6.1
>Reporter: Na Zhai
>Assignee: Na Zhai
>Priority: Major
> Fix For: v2.6.2
>
>
> In my windows env, when I run Kylin, Kylin service can't start. In class
> ZKUtil, I found Kylin use File(path).getCanonicalPath() to norm windows path,
> however this will get a path like ' C:\kylin\kylin_metadata'.For Zookeeper,
> the path must start with / character.
> {color:#FF}return new File(path).toURI().getPath(){color} might be
> better!!!
> Below is stackTrace:
> : java.lang.RuntimeException:
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException: Fail to check or create znode for chRoot
> F:\kylin\kylin_metadata_idea due to
> at org.apache.kylin.common.util.ZKUtil.getZookeeperClient(ZKUtil.java:137)
> at org.apache.kylin.common.util.ZKUtil.getZookeeperClient(ZKUtil.java:115)
> at
> org.apache.kylin.job.lock.zookeeper.ZookeeperDistributedLock$Factory.(ZookeeperDistributedLock.java:57)
> at
> org.apache.kylin.job.lock.zookeeper.ZookeeperDistributedLock$Factory.(ZookeeperDistributedLock.java:53)
> at
> org.apache.kylin.job.lock.zookeeper.ZookeeperJobLock.(ZookeeperJobLock.java:32)
> at
> org.apache.kylin.rest.service.JobService.afterPropertiesSet(JobService.java:132)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)
> ... 61 more
> Caused by: com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException: Fail to check or create znode for chRoot
> F:\kylin\kylin_metadata_idea due to
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2263)
> at com.google.common.cache.LocalCache.get(LocalCache.java:4000)
> at
> com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4789)
> at org.apache.kylin.common.util.ZKUtil.getZookeeperClient(ZKUtil.java:123)
> ... 68 more
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3935) ZKUtil acquire the wrong Zookeeper Path on windows
[
https://issues.apache.org/jira/browse/KYLIN-3935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812212#comment-16812212
]
ASF GitHub Bot commented on KYLIN-3935:
---
nichunen commented on pull request #581: KYLIN-3935 ZKUtil acquire the wrong
Zookeeper Path on windows
URL: https://github.com/apache/kylin/pull/581
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> ZKUtil acquire the wrong Zookeeper Path on windows
> --
>
> Key: KYLIN-3935
> URL: https://issues.apache.org/jira/browse/KYLIN-3935
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.6.1
>Reporter: Na Zhai
>Assignee: Na Zhai
>Priority: Major
> Fix For: v2.6.2
>
>
> In my windows env, when I run Kylin, Kylin service can't start. In class
> ZKUtil, I found Kylin use File(path).getCanonicalPath() to norm windows path,
> however this will get a path like ' C:\kylin\kylin_metadata'.For Zookeeper,
> the path must start with / character.
> {color:#FF}return new File(path).toURI().getPath(){color} might be
> better!!!
> Below is stackTrace:
> : java.lang.RuntimeException:
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException: Fail to check or create znode for chRoot
> F:\kylin\kylin_metadata_idea due to
> at org.apache.kylin.common.util.ZKUtil.getZookeeperClient(ZKUtil.java:137)
> at org.apache.kylin.common.util.ZKUtil.getZookeeperClient(ZKUtil.java:115)
> at
> org.apache.kylin.job.lock.zookeeper.ZookeeperDistributedLock$Factory.(ZookeeperDistributedLock.java:57)
> at
> org.apache.kylin.job.lock.zookeeper.ZookeeperDistributedLock$Factory.(ZookeeperDistributedLock.java:53)
> at
> org.apache.kylin.job.lock.zookeeper.ZookeeperJobLock.(ZookeeperJobLock.java:32)
> at
> org.apache.kylin.rest.service.JobService.afterPropertiesSet(JobService.java:132)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)
> ... 61 more
> Caused by: com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException: Fail to check or create znode for chRoot
> F:\kylin\kylin_metadata_idea due to
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2263)
> at com.google.common.cache.LocalCache.get(LocalCache.java:4000)
> at
> com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4789)
> at org.apache.kylin.common.util.ZKUtil.getZookeeperClient(ZKUtil.java:123)
> ... 68 more
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [kylin] nichunen merged pull request #581: KYLIN-3935 ZKUtil acquire the wrong Zookeeper Path on windows
nichunen merged pull request #581: KYLIN-3935 ZKUtil acquire the wrong Zookeeper Path on windows URL: https://github.com/apache/kylin/pull/581 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (KYLIN-3934) sqoop import param '--null-string' result in null value become blank string in hive table
[
https://issues.apache.org/jira/browse/KYLIN-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wanghao updated KYLIN-3934:
---
Description:
when column value from jdbc is null, sqoop will write it into hive table with
blank string.
eg
jdbc:
A | B
1 | 1
2 | 2
a | null
hive table:
A | B
1 | 1
2 | 2
a |
because of this, when I use count(distinct B), it return 3, not 2, and it can
lead to other problems
{code:java}
String cmd = String.format(Locale.ROOT,
"%s/bin/sqoop import" + generateSqoopConfigArgString()
+ "--connect \"%s\" --driver %s --username %s --password %s --query \"%s AND
\\$CONDITIONS\" "
+ "--target-dir %s/%s --split-by %s --boundary-query \"%s\" --null-string '' "
+ "--fields-terminated-by '%s' --num-mappers %d",
sqoopHome, connectionUrl, driverClass, jdbcUser, jdbcPass, selectSql,
jobWorkingDir, hiveTable,
splitColumn, bquery, filedDelimiter, mapperNum);
{code}
the param '–null=string' should be '
\\N' instead of blank string ''.
I resolved this problem by replace the param. But it needs be configured in
kylin.properties
was:
when column value from jdbc is null, sqoop will write it into hive table with
blank string.
eg
jdbc:
A | B
1 | 1
2 | 2
a | null
hive table:
A | B
1 | 1
2 | 2
a |
because of this, when I use count(distinct B), it return 3, not 2, and it can
lead to other problems
{code:java}
String cmd = String.format(Locale.ROOT,
"%s/bin/sqoop import" + generateSqoopConfigArgString()
+ "--connect \"%s\" --driver %s --username %s --password %s --query \"%s AND
\\$CONDITIONS\" "
+ "--target-dir %s/%s --split-by %s --boundary-query \"%s\" --null-string '' "
+ "--fields-terminated-by '%s' --num-mappers %d",
sqoopHome, connectionUrl, driverClass, jdbcUser, jdbcPass, selectSql,
jobWorkingDir, hiveTable,
splitColumn, bquery, filedDelimiter, mapperNum);
{code}
the param '–null=string' should be '\\N' instead of blank string ''.
I resolved this problem by replace the param. But it needs be configured in
kylin.properties
> sqoop import param '--null-string' result in null value become blank string
> in hive table
> -
>
> Key: KYLIN-3934
> URL: https://issues.apache.org/jira/browse/KYLIN-3934
> Project: Kylin
> Issue Type: Bug
> Components: Others
>Affects Versions: v2.6.0
>Reporter: wanghao
>Priority: Major
> Fix For: v2.6.2
>
>
> when column value from jdbc is null, sqoop will write it into hive table with
> blank string.
> eg
> jdbc:
> A | B
> 1 | 1
> 2 | 2
> a | null
>
> hive table:
> A | B
> 1 | 1
> 2 | 2
> a |
> because of this, when I use count(distinct B), it return 3, not 2, and it can
> lead to other problems
>
>
> {code:java}
> String cmd = String.format(Locale.ROOT,
> "%s/bin/sqoop import" + generateSqoopConfigArgString()
> + "--connect \"%s\" --driver %s --username %s --password %s --query \"%s AND
> \\$CONDITIONS\" "
> + "--target-dir %s/%s --split-by %s --boundary-query \"%s\" --null-string '' "
> + "--fields-terminated-by '%s' --num-mappers %d",
> sqoopHome, connectionUrl, driverClass, jdbcUser, jdbcPass, selectSql,
> jobWorkingDir, hiveTable,
> splitColumn, bquery, filedDelimiter, mapperNum);
> {code}
> the param '–null=string' should be '
> \\N' instead of blank string ''.
> I resolved this problem by replace the param. But it needs be configured in
> kylin.properties
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [kylin] lshmouse commented on issue #557: KYLIN-3913: Remove getAllOutputs api in ExecutableManager to avoid OOM for large metadata
lshmouse commented on issue #557: KYLIN-3913: Remove getAllOutputs api in ExecutableManager to avoid OOM for large metadata URL: https://github.com/apache/kylin/pull/557#issuecomment-480722015 @nichunen Thanks for your reviews. I will update performance data later. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] lshmouse commented on issue #534: KYLIN-3893: Cube build failed for wrong row key column description
lshmouse commented on issue #534: KYLIN-3893: Cube build failed for wrong row key column description URL: https://github.com/apache/kylin/pull/534#issuecomment-480720975 Add row key column encoding validation when saving cube description This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] lshmouse commented on issue #534: KYLIN-3893: Cube build failed for wrong row key column description
lshmouse commented on issue #534: KYLIN-3893: Cube build failed for wrong row key column description URL: https://github.com/apache/kylin/pull/534#issuecomment-480713348 @shaofengshi Thanks for your advice. I will update the patch later. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3928) pushdown only return one column
[ https://issues.apache.org/jira/browse/KYLIN-3928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812170#comment-16812170 ] ASF GitHub Bot commented on KYLIN-3928: --- javalife0312 commented on pull request #576: KYLIN-3928 pushdown only return one column URL: https://github.com/apache/kylin/pull/576 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > pushdown only return one column > --- > > Key: KYLIN-3928 > URL: https://issues.apache.org/jira/browse/KYLIN-3928 > Project: Kylin > Issue Type: Improvement >Reporter: jinguowei >Assignee: jinguowei >Priority: Major > Fix For: v2.6.2 > > > 下推查询，仅仅只返回一列，应该是返回查询的世界列 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] javalife0312 commented on issue #576: KYLIN-3928 pushdown only return one column
javalife0312 commented on issue #576: KYLIN-3928 pushdown only return one column URL: https://github.com/apache/kylin/pull/576#issuecomment-480711569 my test error This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] javalife0312 closed pull request #576: KYLIN-3928 pushdown only return one column
javalife0312 closed pull request #576: KYLIN-3928 pushdown only return one column URL: https://github.com/apache/kylin/pull/576 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
