[jira] [Commented] (KYLIN-5231) RuntimeException in cube build with Spark engine
[ https://issues.apache.org/jira/browse/KYLIN-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17581591#comment-17581591 ] Xiaoxiang Yu commented on KYLIN-5231: - [~sandeepputrevu] Hi, Kylin 3.X doesn't support Spark 3.X(only Spark 2.4.x), and Kylin 4.X doesn't support Spark 3.2.X or higher version(only Spark 2.4.x or Spark 3.1.x). > RuntimeException in cube build with Spark engine > > > Key: KYLIN-5231 > URL: https://issues.apache.org/jira/browse/KYLIN-5231 > Project: Kylin > Issue Type: Bug >Affects Versions: v3.1.2 >Reporter: Sandeep Putrevu >Priority: Major > > Cube build is failing at *Step #7 : Build Cube with Spark* > > Our configuration: > Spark version : 3.2.1 > Hadoop version : 3.1.1.3.1.4.0 > Kylin version : 3.1.2 > Error : > Exception in thread "main" java.lang.RuntimeException: error execute > org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: > scala.reflect.ClassTag$$anon$1 > at > org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42) > at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) > at > org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955) > at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) > at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) > at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) > at > org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.lang.ClassNotFoundException: scala.reflect.ClassTag$$anon$1 > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:418) > at java.lang.ClassLoader.loadClass(ClassLoader.java:351) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:264) > at > org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:142) > at > org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37) > ... 13 more -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5231) RuntimeException in cube build with Spark engine
[ https://issues.apache.org/jira/browse/KYLIN-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17581444#comment-17581444 ] Sandeep Putrevu commented on KYLIN-5231: This is our first attempt after upgrading the Spark to 3.2.1 , earlier it used to work fine with Spark 2.4 > RuntimeException in cube build with Spark engine > > > Key: KYLIN-5231 > URL: https://issues.apache.org/jira/browse/KYLIN-5231 > Project: Kylin > Issue Type: Bug >Affects Versions: v3.1.2 >Reporter: Sandeep Putrevu >Priority: Major > > Cube build is failing at *Step #7 : Build Cube with Spark* > > Our configuration: > Spark version : 3.2.1 > Hadoop version : 3.1.1.3.1.4.0 > Kylin version : 3.1.2 > Error : > Exception in thread "main" java.lang.RuntimeException: error execute > org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: > scala.reflect.ClassTag$$anon$1 > at > org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42) > at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) > at > org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955) > at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) > at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) > at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) > at > org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.lang.ClassNotFoundException: scala.reflect.ClassTag$$anon$1 > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:418) > at java.lang.ClassLoader.loadClass(ClassLoader.java:351) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:264) > at > org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:142) > at > org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37) > ... 13 more -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5231) RuntimeException in cube build with Spark engine
[ https://issues.apache.org/jira/browse/KYLIN-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17581442#comment-17581442 ] liyang commented on KYLIN-5231: --- Most likely a env problem. Is this set of env used to work? Or it is a first attempt? - Spark version : 3.2.1 - Hadoop version : 3.1.1.3.1.4.0 - Kylin version : 3.1.2 > RuntimeException in cube build with Spark engine > > > Key: KYLIN-5231 > URL: https://issues.apache.org/jira/browse/KYLIN-5231 > Project: Kylin > Issue Type: Bug >Affects Versions: v3.1.2 >Reporter: Sandeep Putrevu >Priority: Major > > Cube build is failing at *Step #7 : Build Cube with Spark* > > Our configuration: > Spark version : 3.2.1 > Hadoop version : 3.1.1.3.1.4.0 > Kylin version : 3.1.2 > Error : > Exception in thread "main" java.lang.RuntimeException: error execute > org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: > scala.reflect.ClassTag$$anon$1 > at > org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42) > at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) > at > org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955) > at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) > at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) > at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) > at > org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.lang.ClassNotFoundException: scala.reflect.ClassTag$$anon$1 > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:418) > at java.lang.ClassLoader.loadClass(ClassLoader.java:351) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:264) > at > org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:142) > at > org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37) > ... 13 more -- This message was sent by Atlassian Jira (v8.20.10#820010)
