[jira] [Created] (KYLIN-3607) can't build cube with spark in v2.5.0
ANIL KUMAR created KYLIN-3607: - Summary: can't build cube with spark in v2.5.0 Key: KYLIN-3607 URL: https://issues.apache.org/jira/browse/KYLIN-3607 Project: Kylin Issue Type: Bug Reporter: ANIL KUMAR in Kylin v2.5.0, can't be built cube at step 8 Convert Cuboid Data to HFile, the following is the related exception: ERROR yarn.ApplicationMaster: User class threw exception: java.lang.RuntimeException: error execute org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root cause: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, java.lang.ExceptionInInitializerError at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247) at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194) at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123) at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:99) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Could not create interface org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the hadoop compatibility jar on the classpath? at org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:73) at org.apache.hadoop.hbase.io.MetricsIO.(MetricsIO.java:31) at org.apache.hadoop.hbase.io.hfile.HFile.(HFile.java:192) ... 15 more Caused by: java.util.NoSuchElementException at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:365) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) at java.util.ServiceLoader$1.next(ServiceLoader.java:480) at org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59) ... 17 more -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-2650) Update to Apache Calcite Avatica 1.12.0
[ https://issues.apache.org/jira/browse/KYLIN-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-2650: -- Description: Apache Calcite Avatica 1.17.0 was released mid-July https://sematext.com/opensee/m/Calcite/FR3K9IxYty1a4ECo1?subj=+ANNOUNCE+Apache+Calcite+Avatica+1+12+0+released This issue upgrades Avatica dependency. was: Apache Calcite Avatica 1.12.0 has just been released. https://sematext.com/opensee/m/Calcite/FR3K9IxYty1a4ECo1?subj=+ANNOUNCE+Apache+Calcite+Avatica+1+12+0+released This issue upgrades Avatica dependency. > Update to Apache Calcite Avatica 1.12.0 > --- > > Key: KYLIN-2650 > URL: https://issues.apache.org/jira/browse/KYLIN-2650 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Minor > > Apache Calcite Avatica 1.17.0 was released mid-July > https://sematext.com/opensee/m/Calcite/FR3K9IxYty1a4ECo1?subj=+ANNOUNCE+Apache+Calcite+Avatica+1+12+0+released > This issue upgrades Avatica dependency. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-2650) Update to Apache Calcite Avatica 1.17.0
[ https://issues.apache.org/jira/browse/KYLIN-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-2650: -- Summary: Update to Apache Calcite Avatica 1.17.0 (was: Update to Apache Calcite Avatica 1.12.0) > Update to Apache Calcite Avatica 1.17.0 > --- > > Key: KYLIN-2650 > URL: https://issues.apache.org/jira/browse/KYLIN-2650 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Minor > > Apache Calcite Avatica 1.17.0 was released mid-July > https://sematext.com/opensee/m/Calcite/FR3K9IxYty1a4ECo1?subj=+ANNOUNCE+Apache+Calcite+Avatica+1+12+0+released > This issue upgrades Avatica dependency. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3417) Consider replacing ReentrantReadWriteLock with StampedLock
[
https://issues.apache.org/jira/browse/KYLIN-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16624443#comment-16624443
]
Ted Yu edited comment on KYLIN-3417 at 10/1/18 4:24 PM:
For phase I, we don't need to use Optimistic read lock.
was (Author: yuzhih...@gmail.com):
For phase I, we don't need to use Optimistic read lock
> Consider replacing ReentrantReadWriteLock with StampedLock
> --
>
> Key: KYLIN-3417
> URL: https://issues.apache.org/jira/browse/KYLIN-3417
> Project: Kylin
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Major
> Fix For: Backlog
>
>
> ReentrantReadWriteLock's are only the right solution when there is long hold
> time due to expensive I/O.
> It is expensive for readers.
> We should see if the lighter {{StampedLock}} can be used instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3290) Avoid calling Class#newInstance
[ https://issues.apache.org/jira/browse/KYLIN-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3290: -- Description: Class#newInstance is deprecated starting in Java 9 - https://bugs.openjdk.java.net/browse/JDK-6850612 - because it may throw undeclared checked exceptions. The suggested replacement is getDeclaredConstructor().newInstance(), which wraps the checked exceptions in InvocationException. was: Class#newInstance is deprecated starting in Java 9 - https://bugs.openjdk.java.net/browse/JDK-6850612 - because it may throw undeclared checked exceptions. The suggested replacement is getDeclaredConstructor().newInstance(), which wraps the checked exceptions in InvocationException. > Avoid calling Class#newInstance > --- > > Key: KYLIN-3290 > URL: https://issues.apache.org/jira/browse/KYLIN-3290 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Assignee: jiatao.tao >Priority: Minor > Labels: jdk > Fix For: v2.6.0 > > > Class#newInstance is deprecated starting in Java 9 - > https://bugs.openjdk.java.net/browse/JDK-6850612 - because it may throw > undeclared checked exceptions. > The suggested replacement is getDeclaredConstructor().newInstance(), which > wraps the checked exceptions in InvocationException. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3556) Interned string should not be used as lock object
[
https://issues.apache.org/jira/browse/KYLIN-3556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated KYLIN-3556:
--
Description:
In JDBCResourceDAO :
{code}
public void execute(Connection connection) throws SQLException {
synchronized (resPath.intern()) {
{code}
Locking on an interned string can cause unexpected locking collisions with
other part of code.
was:
In JDBCResourceDAO :
{code}
public void execute(Connection connection) throws SQLException {
synchronized (resPath.intern()) {
{code}
Locking on an interned string can cause unexpected locking collisions with
other part of code.
> Interned string should not be used as lock object
> -
>
> Key: KYLIN-3556
> URL: https://issues.apache.org/jira/browse/KYLIN-3556
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.5.0
>Reporter: Ted Yu
>Assignee: Kaige Liu
>Priority: Minor
> Fix For: v2.5.1
>
>
> In JDBCResourceDAO :
> {code}
> public void execute(Connection connection) throws SQLException {
> synchronized (resPath.intern()) {
> {code}
> Locking on an interned string can cause unexpected locking collisions with
> other part of code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3607) can't build cube with spark in v2.5.0
[ https://issues.apache.org/jira/browse/KYLIN-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16634872#comment-16634872 ] Shaofeng SHI commented on KYLIN-3607: - What's the hadoop and hbase version? > can't build cube with spark in v2.5.0 > - > > Key: KYLIN-3607 > URL: https://issues.apache.org/jira/browse/KYLIN-3607 > Project: Kylin > Issue Type: Bug >Reporter: ANIL KUMAR >Priority: Major > > in Kylin v2.5.0, can't be built cube at step 8 Convert Cuboid Data to HFile, > the following is the related exception: > > ERROR yarn.ApplicationMaster: User class threw exception: > java.lang.RuntimeException: error execute > org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root cause: Job aborted > due to stage failure: Task 0 in stage 1.0 failed 4 times, > java.lang.ExceptionInInitializerError > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247) > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194) > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123) > at > org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) > at org.apache.spark.scheduler.Task.run(Task.scala:99) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Could not create interface > org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the > hadoop compatibility jar on the classpath? > at > org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:73) > at org.apache.hadoop.hbase.io.MetricsIO.(MetricsIO.java:31) > at org.apache.hadoop.hbase.io.hfile.HFile.(HFile.java:192) > ... 15 more > Caused by: java.util.NoSuchElementException > at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:365) > at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) > at java.util.ServiceLoader$1.next(ServiceLoader.java:480) > at > org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59) > ... 17 more -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3607) can't build cube with spark in v2.5.0
[ https://issues.apache.org/jira/browse/KYLIN-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16634875#comment-16634875 ] ANIL KUMAR commented on KYLIN-3607: --- Hadoop 2.6.4.25-1 HBase 1.1.2.2.6.4.25-1 > can't build cube with spark in v2.5.0 > - > > Key: KYLIN-3607 > URL: https://issues.apache.org/jira/browse/KYLIN-3607 > Project: Kylin > Issue Type: Bug >Reporter: ANIL KUMAR >Priority: Major > > in Kylin v2.5.0, can't be built cube at step 8 Convert Cuboid Data to HFile, > the following is the related exception: > > ERROR yarn.ApplicationMaster: User class threw exception: > java.lang.RuntimeException: error execute > org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root cause: Job aborted > due to stage failure: Task 0 in stage 1.0 failed 4 times, > java.lang.ExceptionInInitializerError > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247) > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194) > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123) > at > org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) > at org.apache.spark.scheduler.Task.run(Task.scala:99) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Could not create interface > org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the > hadoop compatibility jar on the classpath? > at > org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:73) > at org.apache.hadoop.hbase.io.MetricsIO.(MetricsIO.java:31) > at org.apache.hadoop.hbase.io.hfile.HFile.(HFile.java:192) > ... 15 more > Caused by: java.util.NoSuchElementException > at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:365) > at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) > at java.util.ServiceLoader$1.next(ServiceLoader.java:480) > at > org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59) > ... 17 more -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-2517) Upgrade hbase dependency to 1.4.7
[ https://issues.apache.org/jira/browse/KYLIN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-2517: -- Summary: Upgrade hbase dependency to 1.4.7 (was: Upgrade hbase dependency to 1.4.6) > Upgrade hbase dependency to 1.4.7 > - > > Key: KYLIN-2517 > URL: https://issues.apache.org/jira/browse/KYLIN-2517 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Major > > There have been major enhancements / bug fixes since the hbase 1.1.1 release. > This issue is to upgrade to 1.4.6 release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-2517) Upgrade hbase dependency to 1.4.7
[ https://issues.apache.org/jira/browse/KYLIN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-2517: -- Description: There have been major enhancements / bug fixes since the hbase 1.1.1 release. This issue is to upgrade to 1.4.7 release. was: There have been major enhancements / bug fixes since the hbase 1.1.1 release. This issue is to upgrade to 1.4.6 release. > Upgrade hbase dependency to 1.4.7 > - > > Key: KYLIN-2517 > URL: https://issues.apache.org/jira/browse/KYLIN-2517 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Major > > There have been major enhancements / bug fixes since the hbase 1.1.1 release. > This issue is to upgrade to 1.4.7 release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3046) Consider introducing log4j-extras
[ https://issues.apache.org/jira/browse/KYLIN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3046: -- Description: log4j-extras allows log rotation as well as compression. https://logging.apache.org/log4j/extras/download.html We should consider using log4j-extras. was: log4j-extras allows log rotation as well as compression. https://logging.apache.org/log4j/extras/download.html We should consider using log4j-extras. > Consider introducing log4j-extras > -- > > Key: KYLIN-3046 > URL: https://issues.apache.org/jira/browse/KYLIN-3046 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Major > Labels: log > Fix For: Backlog > > > log4j-extras allows log rotation as well as compression. > https://logging.apache.org/log4j/extras/download.html > We should consider using log4j-extras. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
