Thanks to both of you for your advice on this. I tried out with command line but its getting error out – Pls. advise. From Jar file kylin-job-2.1.0.jar, These class file needs to be read.
SparkCubingByLayer SparkEntry /apps/rft/rcmo/apps/kylin/kylin_namespace/apache-kylin-2.1.0-KYLIN-2846-cdh57/spark/bin/spark-submit --class org.apache.kylin.common.util.SparkEntry --conf spark.history.fs.logDirectory=hdfs://sfpdev/tenants/rft/rcmo/kylin/spark-history --conf spark.eventLog.dir=hdfs://sfpdev/tenants/rft/rcmo/kylin/spark-history --conf spark.yarn.queue=RCMO_Pool --conf spark.master=yarn --jars /apps/rft/rcmo/apps/kylin/kylin_namespace/apache-kylin-2.1.0-KYLIN-2846-cdh57/lib/kylin-job-2.1.0.jar -className org.apache.kylin.engine.spark.SparkCubingByLayer -hiveTable db_rft_rcmo_rfda.kylin_intermediate_drr_cube_saprk_2_b9d800a6_5e7e_46dc_847b_b2925369b574 -segmentId b9d800a6-5e7e-46dc-847b-b2925369b574 -confPath /apps/rft/rcmo/apps/kylin/kylin_namespace/apache-kylin-2.1.0-KYLIN-2846-cdh57/conf -output hdfs://sfpdev/tenants/rft/rcmo/kylin/ns_rft_rcmo_creg_poc-kylin_metadata/kylin-65963dfd-393e-4198-9dd2-51d54a6bf6c4/DRR_CUBE_SAPRK_2/cuboid/ -cubename DRR_CUBE_SAPRK_2 Error: Unrecognized option: -className Regards, Manoj From: prasanna lakshmi [mailto:prasannapadar...@gmail.com] Sent: Saturday, December 02, 2017 10:29 PM To: user@kylin.apache.org Subject: Re: Apache kylin 2.1 on Spark Hi Manoj As part of my experience try the same command in prompt by using configuration spark in yarn mode only. Remove all other conf settings in that command. if it is successful then you comment all those properties in kylin.properties file of kylin/conf folder. On Nov 30, 2017 3:01 PM, "Kumar, Manoj H" <manoj.h.ku...@jpmorgan.com<mailto:manoj.h.ku...@jpmorgan.com>> wrote: Pls. advise on this as I am running Cube building process using Spark Engine. What setting is missing here? kylin.env.hadoop-conf-dir=/etc/hive/conf 224 # 225 ## Estimate the RDD partition numbers 226 kylin.engine.spark.rdd-partition-cut-mb=100 227 # 228 ## Minimal partition numbers of rdd 229 kylin.engine.spark.min-partition=1 230 # 231 ## Max partition numbers of rdd 232 kylin.engine.spark.max-partition=5000 233 # 234 ## Spark conf (default is in spark/conf/spark-defaults.conf) 235 kylin.engine.spark-conf.spark.master=yarn 236 kylin.engine.spark-conf.spark.submit.deployMode=cluster 237 kylin.engine.spark-conf.spark.yarn.queue=RCMO_Pool 238 kylin.engine.spark-conf.spark.executor.memory=4G 239 kylin.engine.spark-conf.spark.executor.cores=2 240 kylin.engine.spark-conf.spark.executor.instances=1 241 kylin.engine.spark-conf.spark.eventLog.enabled=true 242 kylin.engine.spark-conf.spark.eventLog.dir=hdfs\:///kylin/spark-history 243 kylin.engine.spark-conf.spark.history.fs.logDirectory=hdfs\:///kylin/spark-history 244 kylin.engine.spark-conf.spark.hadoop.yarn.timeline-service.enabled=false 245 # 246 ## manually upload spark-assembly jar to HDFS and then set this property will avoid repeatedly uploading jar at runtime 247 ##kylin.engine.spark-conf.spark.yarn.jar=hdfs://namenode:8020/kylin/spark/spark-assembly-1.6.3-hadoop2.6.0.jar 248 kylin.engine.spark-conf.spark.io.compression.codec=org.apache.spark.io<http://apache.spark.io>.SnappyCompressionCodec 2017-11-30 04:16:50,156 ERROR [Job 50b5d7ce-35e6-438d-94f9-0b969adfc1bb-192] spark.SparkExecutable:133 : error run spark job: 5207 java.io.IOException: OS command error exit with 1 -- export HADOOP_CONF_DIR=/etc/hive/conf && /apps/rft/rcmo/apps/kylin/kylin_namespace/apache-kylin-2.1.0-KYLIN-2846-cdh57/spark/bin/sp ark-submit --class org.apache.kylin.common.util.SparkEntry --conf spark.executor.instances=1 --conf spark.yarn.queue=RCMO_Pool --conf spark.history.fs.logDirectory=hdfs:///kylin/spa rk-history --conf spark.io.compression.codec=org.apache.spark.io<http://org.apache.spark.io>.SnappyCompressionCodec --conf spark.master=yarn --conf spark.hadoop.yarn.timeline-service.enabled=false --conf spar k.executor.memory=4G --conf spark.eventLog.enabled=true --conf spark.eventLog.dir=hdfs:///kylin/spark-history --conf spark.executor.cores=2 --conf spark.submit.deployMode=cluster - -files /etc/hbase/conf.cloudera.hbase/hbase-site.xml --jars /apps/rft/rcmo/apps/kylin/kylin_namespace/apache-kylin-2.1.0-KYLIN-2846-cdh57/spark/jars/htrace-core-3.0.4.jar,/opt/cloudera /parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/jars/htrace-core-3.2.0-incubating.jar,/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/jars/hbase-client-1.2.0-cdh5.9.1.jar,/opt/cloudera/parcels/CDH- 5.9.1-1.cdh5.9.1.p0.4/jars/hbase-common-1.2.0-cdh5.9.1.jar,/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/jars/hbase-protocol-1.2.0-cdh5.9.1.jar,/opt/cloudera/parcels/CDH-5.9.1-1.cdh5 .9.1.p0.4/jars/metrics-core-2.2.0.jar,/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/jars/guava-12.0.1.jar, /apps/rft/rcmo/apps/kylin/kylin_namespace/apache-kylin-2.1.0-KYLIN-2846-cdh 57/lib/kylin-job-2.1.0.jar -className org.apache.kylin.engine.spark.SparkCubingByLayer -hiveTable db_rft_rcmo_rfda.kylin_intermediate_drr_cube_saprk_15128ffc_67b5_476b_a942_48645346b64 f -segmentId 15128ffc-67b5-476b-a942-48645346b64f -confPath /apps/rft/rcmo/apps/kylin/kylin_namespace/apache-kylin-2.1.0-KYLIN-2846-cdh57/conf -output hdfs://sfpdev/tenants/rft/rcmo/ky lin/ns_rft_rcmo_creg_poc-kylin_metadata/kylin-50b5d7ce-35e6-438d-94f9-0b969adfc1bb/DRR_CUBE_SAPRK/cuboid/ -cubename DRR_CUBE_SAPRK 5208 17/11/30 04:16:10 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm76 5209 17/11/30 04:16:11 INFO yarn.Client: Requesting a new application from cluster with 19 NodeManagers 5210 17/11/30 04:16:11 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (272850 MB per container) 5211 17/11/30 04:16:11 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead 5212 17/11/30 04:16:11 INFO yarn.Client: Setting up container launch context for our AM 5213 17/11/30 04:16:11 INFO yarn.Client: Setting up the launch environment for our AM container 5214 17/11/30 04:16:11 INFO yarn.Client: Preparing resources for our AM container 5215 17/11/30 04:16:11 INFO security.HDFSCredentialProvider: getting token for namenode: hdfs://sfpdev/user/a_rcmo_nd 5216 17/11/30 04:16:11 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 704970 for a_rcmo_nd on ha-hdfs:sfpdev 5217 17/11/30 04:16:14 INFO hive.metastore: Trying to connect to metastore with URI thrift://bdtpisr3n1.svr.us.jpmchase.net:9083<http://bdtpisr3n1.svr.us.jpmchase.net:9083> 5218 17/11/30 04:16:14 INFO hive.metastore: Connected to metastore. 5219 17/11/30 04:16:15 WARN token.Token: Cannot find class for token kind HIVE_DELEGATION_TOKEN 5220 17/11/30 04:16:15 INFO security.HiveCredentialProvider: Get Token from hive metastore: Kind: HIVE_DELEGATION_TOKEN, Service: , Ident: 00 25 61 5f 72 63 6d 6f 5f 6e 64 40 4e 41 45 41 53 54 2e 41 44 2e 4a 50 4d 4f 52 47 41 4e 43 48 41 53 45 2e 43 4f 4d 04 68 69 76 65 00 8a 01 60 0c 36 56 a7 8a 01 60 30 42 da a7 8e 39 8a 22 5221 17/11/30 04:16:15 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. 5222 17/11/30 04:16:19 INFO yarn.Client: Uploading resource file:/tmp/spark-8cfdcc86-1bf2-4baf-b4c1-4f776c708490/__spark_libs__4437186145258387608.zip -> hdfs://sfpdev/user/a_rcmo_nd/.spark Staging/application_1509132635807_32198/__spark_libs__4437186145258387608.zip 5223 17/11/30 04:16:21 INFO yarn.Client: Uploading resource file:/apps/rft/rcmo/apps/kylin/kylin_namespace/apache-kylin-2.1.0-KYLIN-2846-cdh57/lib/kylin-job-2.1.0.jar -> hdfs://sfpdev/user/ a_rcmo_nd/.sparkStaging/application_1509132635807_32198/kylin-job-2.1.0.jar 5224 17/11/30 04:16:21 INFO yarn.Client: Uploading resource file:/apps/rft/rcmo/apps/kylin/kylin_namespace/apache-kylin-2.1.0-KYLIN-2846-cdh57/spark/jars/htrace-core-3.0.4.jar -> hdfs://sfp dev/user/a_rcmo_nd/.sparkStaging/application_1509132635807_32198/htrace-core-3.0.4.jar 5225 17/11/30 04:16:21 INFO yarn.Client: Uploading resource file:/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/jars/htrace-core-3.2.0-incubating.jar -> hdfs://sfpdev/user/a_rcmo_nd/.spark Staging/application_1509132635807_32198/htrace-core-3.2.0-incubating.jar 5226 17/11/30 04:16:21 INFO yarn.Client: Uploading resource file:/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/jars/hbase-client-1.2.0-cdh5.9.1.jar -> hdfs://sfpdev/user/a_rcmo_nd/.sparkS taging/application_1509132635807_32198/hbase-client-1.2.0-cdh5.9.1.jar 5227 17/11/30 04:16:21 INFO yarn.Client: Uploading resource file:/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/jars/hbase-common-1.2.0-cdh5.9.1.jar -> hdfs://sfpdev/user/a_rcmo_nd/.sparkS taging/application_1509132635807_32198/hbase-common-1.2.0-cdh5.9.1.jar 5228 17/11/30 04:16:22 INFO yarn.Client: Uploading resource file:/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/jars/hbase-protocol-1.2.0-cdh5.9.1.jar -> hdfs://sfpdev/user/a_rcmo_nd/.spar kStaging/application_1509132635807_32198/hbase-protocol-1.2.0-cdh5.9.1.jar 5229 17/11/30 04:16:22 INFO yarn.Client: Uploading resource file:/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/jars/metrics-core-2.2.0.jar -> hdfs://sfpdev/user/a_rcmo_nd/.sparkStaging/ap plication_1509132635807_32198/metrics-core-2.2.0.jar 9 17/11/30 04:16:22 INFO yarn.Client: Uploading resource file:/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/jars/metrics-core-2.2.0.jar -> hdfs://sfpdev/user/a_rcmo_nd/.sparkStaging/ap plication_1509132635807_32198/metrics-core-2.2.0.jar 5230 17/11/30 04:16:22 INFO yarn.Client: Uploading resource file:/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/jars/guava-12.0.1.jar -> hdfs://sfpdev/user/a_rcmo_nd/.sparkStaging/applicat ion_1509132635807_32198/guava-12.0.1.jar 5231 17/11/30 04:16:22 INFO yarn.Client: Uploading resource file:/etc/hbase/conf.cloudera.hbase/hbase-site.xml -> hdfs://sfpdev/user/a_rcmo_nd/.sparkStaging/application_1509132635807_32198/ hbase-site.xml 5232 17/11/30 04:16:22 INFO yarn.Client: Uploading resource file:/tmp/spark-8cfdcc86-1bf2-4baf-b4c1-4f776c708490/__spark_conf__5140031382599260375.zip -> hdfs://sfpdev/user/a_rcmo_nd/.spark Staging/application_1509132635807_32198/__spark_conf__.zip 5233 17/11/30 04:16:22 INFO spark.SecurityManager: Changing view acls to: a_rcmo_nd 5234 17/11/30 04:16:22 INFO spark.SecurityManager: Changing modify acls to: a_rcmo_nd 5235 17/11/30 04:16:22 INFO spark.SecurityManager: Changing view acls groups to: 5236 17/11/30 04:16:22 INFO spark.SecurityManager: Changing modify acls groups to: 5237 17/11/30 04:16:22 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(a_rcmo_nd); groups with view permissions: Se t(); users with modify permissions: Set(a_rcmo_nd); groups with modify permissions: Set() 5238 17/11/30 04:16:22 INFO yarn.Client: Submitting application application_1509132635807_32198 to ResourceManager 5239 17/11/30 04:16:22 INFO impl.YarnClientImpl: Submitted application application_1509132635807_32198 5240 17/11/30 04:16:23 INFO yarn.Client: Application report for application_1509132635807_32198 (state: ACCEPTED) 5241 17/11/30 04:16:23 INFO yarn.Client: 5242 client token: Token { kind: YARN_CLIENT_TOKEN, service: } 5243 diagnostics: N/A 5244 ApplicationMaster host: N/A 5245 ApplicationMaster RPC port: -1 5246 queue: root.RCMO_Pool 5247 start time: 1512033382431 5248 final status: UNDEFINED 5249 tracking URL: http://bdtpisr3n2.svr.us.jpmchase.net:8088/proxy/application_1509132635807_32198/ 5250 user: a_rcmo_nd 5251 17/11/30 04:16:24 INFO yarn.Client: Application report for application_1509132635807_32198 (state: ACCEPTED) 5252 17/11/30 04:16:25 INFO yarn.Client: Application report for application_1509132635807_32198 (state: ACCEPTED) 5253 17/11/30 04:16:26 INFO yarn.Client: Application report for application_1509132635807_32198 (state: ACCEPTED) 5254 17/11/30 04:16:27 INFO yarn.Client: Application report for application_1509132635807_32198 (state: ACCEPTED) 5255 17/11/30 04:16:28 INFO yarn.Client: Application report for application_1509132635807_32198 (state: ACCEPTED) 5256 17/11/30 04:16:29 INFO yarn.Client: Application report for application_1509132635807_32198 (state: ACCEPTED) 5257 17/11/30 04:16:30 INFO yarn.Client: Application report for application_1509132635807_32198 (state: ACCEPTED) 5258 17/11/30 04:16:31 INFO yarn.Client: Application report for application_1509132635807_32198 (state: ACCEPTED) 5259 17/11/30 04:16:32 INFO yarn.Client: Application report for application_1509132635807_32198 (state: ACCEPTED) 5260 17/11/30 04:16:33 INFO yarn.Client: Application report for application_1509132635807_32198 (state: ACCEPTED) 5261 17/11/30 04:16:34 INFO yarn.Client: Application report for application_1509132635807_32198 (state: ACCEPTED) 5262 17/11/30 04:16:35 INFO yarn.Client: Application report for application_1509132635807_32198 (state: ACCEPTED) 5263 17/11/30 04:16:36 INFO yarn.Client: Application report for application_1509132635807_32198 (state: ACCEPTED) 5264 17/11/30 04:16:37 INFO yarn.Client: Application report for application_1509132635807_32198 (state: ACCEPTED) 5265 17/11/30 04:16:38 INFO yarn.Client: Application report for application_1509132635807_32198 (state: ACCEPTED) 5266 17/11/30 04:16:39 INFO yarn.Client: Application report for application_1509132635807_32198 (state: ACCEPTED) 5276 17/11/30 04:16:49 INFO yarn.Client: Application report for application_1509132635807_32198 (state: FAILED) 5277 17/11/30 04:16:49 INFO yarn.Client: 5278 client token: N/A 5279 diagnostics: Application application_1509132635807_32198 failed 2 times due to AM Container for appattempt_1509132635807_32198_000002 exited with exitCode: 15 5280 For more detailed output, check application tracking page:http://bdtpisr3n2.svr.us.jpmchase.net:8088/proxy/application_1509132635807_32198/Then, click on links to logs of each attempt. 5281 Diagnostics: Exception from container-launch. 5282 Container id: container_e113_1509132635807_32198_02_000001 5283 Exit code: 15 5284 Stack trace: ExitCodeException exitCode=15: 5285 at org.apache.hadoop.util.Shell.runCommand(Shell.java:601) 5286 at org.apache.hadoop.util.Shell.run(Shell.java:504) 5287 at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786) 5288 at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:373) 5289 at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) 5290 at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) 5291 at java.util.concurrent.FutureTask.run(FutureTask.java:266) 5292 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 5293 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 5294 at java.lang.Thread.run(Thread.java:745) 5295 5296 Shell output: main : command provided 1 Regards, Manoj This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited. This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
