A solution is to put a "java-opts" file in spark/conf folder, adding the 'hdp.version' configuration, like this:
cat /usr/local/spark/conf/java-opts -Dhdp.version=2.4.0.0-169 Best regards, Shaofeng Shi 史少锋 Apache Kylin PMC Work email: [email protected] Kyligence Inc: https://kyligence.io/ Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html Join Kylin user mail group: [email protected] Join Kylin dev mail group: [email protected] Kang-Sen Lu <[email protected]> 于2018年11月30日周五 下午9:04写道： > Thanks for the reply from Yichen and Aron. This is my kylin.properties: > > > > kylin.engine.spark-conf.spark.yarn.archive=hdfs:// > 192.168.230.199:8020/user/zettics/spark/spark-libs.jar > > > ##kylin.engine.spark-conf.spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec > > # > > ## uncomment for HDP > > > kylin.engine.spark-conf.spark.driver.extraJavaOptions=-Dhdp.version=2.5.6.0-40 > > > kylin.engine.spark-conf.spark.yarn.am.extraJavaOptions=-Dhdp.version=2.5.6.0-40 > > > kylin.engine.spark-conf.spark.executor.extraJavaOptions=-Dhdp.version=2.5.6.0-40 > > > > But I still get the same error. > > > > Stack trace: ExitCodeException exitCode=1: > /data5/hadoop/yarn/local/usercache/zettics/appcache/application_1543422353836_0091/container_e05_1543422353836_0091_02_000001/launch_container.sh: > line 26: > $PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:$HADOOP_CONF_DIR:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure: > bad substitution > > > > at org.apache.hadoop.util.Shell.runCommand(Shell.java:944) > > at org.apache.hadoop.util.Shell.run(Shell.java:848) > > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1142) > > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) > > > > I also saw in stderr: > > > > Log Type: stderr > > Log Upload Time: Fri Nov 30 07:54:45 -0500 2018 > > Log Length: 88 > > Error: Could not find or load main class > org.apache.spark.deploy.yarn.ApplicationMaster > > > > I suspect my problem is related to the fact that “${hdp.version}” was not > resolved somehow. It seems that kylin.properties parameters like > “extraJavaOptions=-Dhdp.version=2.5.6.0-40” was not enough. > > > > Kang-sen > > > > > > > > > > > > *From:* Yichen Zhou <[email protected]> > *Sent:* Thursday, November 29, 2018 9:08 PM > *To:* [email protected] > *Subject:* Re: anybody used spark to build cube in kylin 2.5.1? > > > > Hi Kang-Sen, > > > > I think Jiatao is right. If you want to use spark to build cube in HDP > cluster, you need to config -Dhdp.version in > $KYLIN_HOME/conf/kylin.properties. > > ## uncomment for HDP > > #kylin.engine.spark-conf.spark.driver.extraJavaOptions=-Dhdp.version=current > > #kylin.engine.spark-conf.spark.yarn.am.extraJavaOptions=-Dhdp.version=current > > #kylin.engine.spark-conf.spark.executor.extraJavaOptions=-Dhdp.version=current > > Please refer to this: > http://kylin.apache.org/docs/tutorial/cube_spark.html > > > > Regards, > > Yichen > > > > > > JiaTao Tao <[email protected]> 于2018年11月30日周五 上午9:57写道： > > Hi > > > > I took a look at the Internet and found these links, take a try and hope > it helps. > > > > > https://community.hortonworks.com/questions/23699/bad-substitution-error-running-spark-on-yarn.html > > > > > https://stackoverflow.com/questions/32341709/bad-substitution-when-submitting-spark-job-to-yarn-cluster > > > > -- > > > > Regards! > > Aron Tao > > > > > > Kang-Sen Lu <[email protected]> 于2018年11月29日周四 下午3:11写道： > > We are running kylin 2.5.1. For a specific cube created, the cube build > for one hour of data took 200 minutes. So I am thinking about building cube > with spark, instead of map-reduce. > > > > I selected spark in the cube design, advanced setting. > > > > The cube build failed at step 3, with the following error log: > > > > OS command error exit with return code: 1, error message: 18/11/29 > 09:50:33 INFO client.RMProxy: Connecting to ResourceManager at > anovadata6.anovadata.local/192.168.230.199:8050 > > 18/11/29 09:50:33 INFO yarn.Client: Requesting a new application from > cluster with 1 NodeManagers > > 18/11/29 09:50:33 INFO yarn.Client: Verifying our application has not > requested more than the maximum memory capability of the cluster (191488 MB > per container) > > 18/11/29 09:50:33 INFO yarn.Client: Will allocate AM container, with 2432 > MB memory including 384 MB overhead > > 18/11/29 09:50:33 INFO yarn.Client: Setting up container launch context > for our AM > > 18/11/29 09:50:33 INFO yarn.Client: Setting up the launch environment for > our AM container > > 18/11/29 09:50:33 INFO yarn.Client: Preparing resources for our AM > container > > 18/11/29 09:50:35 WARN yarn.Client: Neither spark.yarn.jars nor > spark.yarn.archive is set, falling back to uploading libraries under > SPARK_HOME. > > 18/11/29 09:50:38 INFO yarn.Client: Uploading resource > file:/tmp/spark-507691d4-f131-4bc5-bf6c-c8ff7606e201/__spark_libs__6261254232609828730.zip > -> > hdfs://anovadata6.anovadata.local:8020/user/zettics/.sparkStaging/application_1543422353836_0088/__spark_libs__6261254232609828730.zip > > 18/11/29 09:50:39 INFO yarn.Client: Uploading resource > file:/home/zettics/kylin/apache-kylin-2.5.1-anovadata-bin/lib/kylin-job-2.5.1-anovadata.jar > -> > hdfs://anovadata6.anovadata.local:8020/user/zettics/.sparkStaging/application_1543422353836_0088/kylin-job-2.5.1-anovadata.jar > > 18/11/29 09:50:39 WARN yarn.Client: Same path resource > file:/home/zettics/kylin/apache-kylin-2.5.1-anovadata-bin/lib/kylin-job-2.5.1-anovadata.jar > added multiple times to distributed cache. > > 18/11/29 09:50:39 INFO yarn.Client: Uploading resource > file:/tmp/spark-507691d4-f131-4bc5-bf6c-c8ff7606e201/__spark_conf__1525388499029792228.zip > -> > hdfs://anovadata6.anovadata.local:8020/user/zettics/.sparkStaging/application_1543422353836_0088/__spark_conf__.zip > > 18/11/29 09:50:39 WARN yarn.Client: spark.yarn.am.extraJavaOptions will > not take effect in cluster mode > > 18/11/29 09:50:39 INFO spark.SecurityManager: Changing view acls to: > zettics > > 18/11/29 09:50:39 INFO spark.SecurityManager: Changing modify acls to: > zettics > > 18/11/29 09:50:39 INFO spark.SecurityManager: Changing view acls groups > to: > > 18/11/29 09:50:39 INFO spark.SecurityManager: Changing modify acls groups > to: > > 18/11/29 09:50:39 INFO spark.SecurityManager: SecurityManager: > authentication disabled; ui acls disabled; users with view permissions: > Set(zettics); groups with view permissions: Set(); users with modify > permissions: Set(zettics); groups with modify permissions: Set() > > 18/11/29 09:50:39 INFO yarn.Client: Submitting application > application_1543422353836_0088 to ResourceManager > > 18/11/29 09:50:39 INFO impl.YarnClientImpl: Submitted application > application_1543422353836_0088 > > 18/11/29 09:50:40 INFO yarn.Client: Application report for > application_1543422353836_0088 (state: ACCEPTED) > > 18/11/29 09:50:40 INFO yarn.Client: > > client token: N/A > > diagnostics: AM container is launched, waiting for AM container to > Register with RM > > ApplicationMaster host: N/A > > ApplicationMaster RPC port: -1 > > queue: default > > start time: 1543503039903 > > final status: UNDEFINED > > tracking URL: > http://anovadata6.anovadata.local:8088/proxy/application_1543422353836_0088/ > > user: zettics > > 18/11/29 09:50:41 INFO yarn.Client: Application report for > application_1543422353836_0088 (state: ACCEPTED) > > 18/11/29 09:50:42 INFO yarn.Client: Application report for > application_1543422353836_0088 (state: ACCEPTED) > > 18/11/29 09:50:43 INFO yarn.Client: Application report for > application_1543422353836_0088 (state: FAILED) > > 18/11/29 09:50:43 INFO yarn.Client: > > client token: N/A > > diagnostics: Application application_1543422353836_0088 failed 2 > times due to AM Container for appattempt_1543422353836_0088_000002 exited > with exitCode: 1 > > For more detailed output, check the application tracking page: > http://anovadata6.anovadata.local:8088/cluster/app/application_1543422353836_0088 > Then click on links to logs of each attempt. > > Diagnostics: Exception from container-launch. > > Container id: container_e05_1543422353836_0088_02_000001 > > Exit code: 1 > > Exception message: > /hadoop/yarn/local/usercache/zettics/appcache/application_1543422353836_0088/container_e05_1543422353836_0088_02_000001/launch_container.sh: > line 26: > $PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:$HADOOP_CONF_DIR:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure: > bad substitution > > > > Stack trace: ExitCodeException exitCode=1: > /hadoop/yarn/local/usercache/zettics/appcache/application_1543422353836_0088/container_e05_1543422353836_0088_02_000001/launch_container.sh: > line 26: > $PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:$HADOOP_CONF_DIR:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure: > bad substitution > > > > at org.apache.hadoop.util.Shell.runCommand(Shell.java:944) > > at org.apache.hadoop.util.Shell.run(Shell.java:848) > > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1142) > > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:317) > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:83) > > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) > > > > > > Thanks. > > > > Kang-sen > > > > >
