[ https://issues.apache.org/jira/browse/OOZIE-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16251571#comment-16251571 ]
Rohini Palaniswamy commented on OOZIE-3122: ------------------------------------------- Just started a discussion on dev mailing list on doing a Apache 4.3.1 release. Hopefully we will have a release in another month. > Oozie spark action unable to handle some options in spark-defaults.conf > ----------------------------------------------------------------------- > > Key: OOZIE-3122 > URL: https://issues.apache.org/jira/browse/OOZIE-3122 > Project: Oozie > Issue Type: Bug > Components: action > Affects Versions: 4.3.0 > Environment: Operating on EMR cluster (5.4.0) which includes Oozie > 4.3.0 > Reporter: Simon Whitelaw > Labels: usability > Original Estimate: 96h > Remaining Estimate: 96h > > When using the setting > oozie.service.SparkConfigurationService.spark.configurations *=etc/spark/conf > to specify a spark-defaults.conf file for spark to use, a few options are not > handled properly by the oozie spark action and cause the job to fail. These > include the following: > * spark.driver.extraClassPath - causes a blank --conf tag to be sent to the > spark submit and the spark submit will fail > * spark.executor.extraClassPath - causes a blank --conf tag to be sent to the > spark submit and the spark submit will fail > * spark.executor.extraJavaOptions with multiple options set (e.g. > spark.executor.extraJavaOptions -verbose:gc -XX:+PrintGCDetails > -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC > -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 > -XX:+CMSClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p') - causes > all options except for the first to be passed as singletons rather than > extraJavaOptions > * spark.driver.extraJavaOptions with multiple option set - causes all options > except for the first to be passed as singletons rather than extraJavaOptions > * Within the spark.driver.extraJavaOptions and > spark.executor.extraJavaOptions, the option -XX:OnOutOfMemoryError='kill -9 > %p' is not parsed correctly even from the spark-opts tag and requires > additional double quotations within the single quote marks: '"kill -9 %p"' > These issues were noticed running the oozie spark action on an amazon EMR > cluster. Workaround involves sending the options as part of the spark-opts > tag, but is not ideal because in EMR, the spark-defaults.conf file is > autogenerated for the specifications of the cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)