[jira] [Commented] (OOZIE-3492) [spark-action] Missing HADOOP_CONF_DIR property
[
https://issues.apache.org/jira/browse/OOZIE-3492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860634#comment-16860634
]
Julia Kinga Marton commented on OOZIE-3492:
---
Thank you [~asalamon74] for the contribution! Is not the nicest one, but I
don't have any other better ideas. +1, committed to master
> [spark-action] Missing HADOOP_CONF_DIR property
> ---
>
> Key: OOZIE-3492
> URL: https://issues.apache.org/jira/browse/OOZIE-3492
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Attachments: OOZIE-3492-001.patch
>
>
> Spark action fails with the following error:
> {noformat}
> Failing Oozie Launcher, When running with master 'yarn-cluster' either
> HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment.
> org.apache.spark.SparkException: When running with master 'yarn-cluster'
> either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment.
> at
> org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:657)
> at
> org.apache.spark.deploy.SparkSubmitArguments.validateSubmitArguments(SparkSubmitArguments.scala:290)
> at
> org.apache.spark.deploy.SparkSubmitArguments.validateArguments(SparkSubmitArguments.scala:251)
> at
> org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:120)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$1.(SparkSubmit.scala:926)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:926)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:81)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:939)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:948)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> at
> org.apache.oozie.action.hadoop.SparkMain.runSpark(SparkMain.java:186)
> at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:93)
> at
> org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:104)
> at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:60)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:410)
> at
> org.apache.oozie.action.hadoop.LauncherAM.access$300(LauncherAM.java:55)
> at
> org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:223)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:217)
> at
> org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141){noformat}
> if the followings are true:
> * yarn.nodemanager.env-whitelist does not contain {{HADOOP_CONF_DIR}}
> * Hadoop version >= 3.1 (or YARN-7677 is backported)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3492) [spark-action] Missing HADOOP_CONF_DIR property
[
https://issues.apache.org/jira/browse/OOZIE-3492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16857490#comment-16857490
]
Hadoop QA commented on OOZIE-3492:
--
Testing JIRA OOZIE-3492
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [6] new bugs found below threshold in total that
must be fixed.
.{color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [webapp].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [server].
.{color:red}-1{color} There are [6] new bugs found below threshold in
[core] that must be fixed, listing only the first [5] ones.
.You can find the SpotBugs diff here (look for the red and orange ones):
core/findbugs-new.html
.The top [5] most important SpotBugs errors are:
.At BulkJPAExecutor.java:[line 207]: This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection
.At BulkJPAExecutor.java:[line 177]: At BulkJPAExecutor.java:[line 176]
.At BulkJPAExecutor.java:[line 206]: At BulkJPAExecutor.java:[line 200]
.This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection: At BulkJPAExecutor.java:[line 207]
.At BulkJPAExecutor.java:[line 112]: At BulkJPAExecutor.java:[line 128]
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in [sharelib/git].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 3169
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/1140/
> [spark-action] Missing HADOOP_CONF_DIR property
> ---
>
> Key: OOZIE-3492
> URL: https://issues.apache.org/jira/browse/OOZIE-3492
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Attachments: OOZIE-3492-001.patch
>
>
> Spark action fails with the following error:
> {noformat}
> Failing Oozie Launcher, When running with master 'yarn-cluster' either
> HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment.
> org.apache.spark.SparkException: When running with master 'yarn-cluster'
> either HADOOP_CONF_DIR or
[jira] [Commented] (OOZIE-3492) [spark-action] Missing HADOOP_CONF_DIR property
[
https://issues.apache.org/jira/browse/OOZIE-3492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16857405#comment-16857405
]
Hadoop QA commented on OOZIE-3492:
--
PreCommit-OOZIE-Build started
> [spark-action] Missing HADOOP_CONF_DIR property
> ---
>
> Key: OOZIE-3492
> URL: https://issues.apache.org/jira/browse/OOZIE-3492
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Attachments: OOZIE-3492-001.patch
>
>
> Spark action fails with the following error:
> {noformat}
> Failing Oozie Launcher, When running with master 'yarn-cluster' either
> HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment.
> org.apache.spark.SparkException: When running with master 'yarn-cluster'
> either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment.
> at
> org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:657)
> at
> org.apache.spark.deploy.SparkSubmitArguments.validateSubmitArguments(SparkSubmitArguments.scala:290)
> at
> org.apache.spark.deploy.SparkSubmitArguments.validateArguments(SparkSubmitArguments.scala:251)
> at
> org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:120)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$1.(SparkSubmit.scala:926)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:926)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:81)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:939)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:948)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> at
> org.apache.oozie.action.hadoop.SparkMain.runSpark(SparkMain.java:186)
> at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:93)
> at
> org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:104)
> at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:60)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:410)
> at
> org.apache.oozie.action.hadoop.LauncherAM.access$300(LauncherAM.java:55)
> at
> org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:223)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:217)
> at
> org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141){noformat}
> if the followings are true:
> * yarn.nodemanager.env-whitelist does not contain {{HADOOP_CONF_DIR}}
> * Hadoop version >= 3.1 (or YARN-7677 is backported)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
