[jira] [Created] (OOZIE-2539) Incorrect property key is used for 'hive log4j configuration file for execution mode'

[abhishek bafna (JIRA)]

abhishek bafna created OOZIE-2539:
-

 Summary: Incorrect property key is used for 'hive log4j 
configuration file for execution mode'
 Key: OOZIE-2539
 URL: https://issues.apache.org/jira/browse/OOZIE-2539
 Project: Oozie
  Issue Type: Bug
Reporter: abhishek bafna
Assignee: abhishek bafna


The correct property key for hive configuration file for execution mode is 
*hive.exec.log4j.file* instead we are setting *hive.log4j.exec.file*. We should 
correct this.

This is working for now, because, If the property is not set, then logging will 
be initialized using *hive-exec-log4j.properties* found on the classpath.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OOZIE-2538) Update HttpClient versions to close security vulnerabilities

[abhishek bafna (JIRA)]

abhishek bafna created OOZIE-2538:
-

 Summary: Update HttpClient versions to close security 
vulnerabilities
 Key: OOZIE-2538
 URL: https://issues.apache.org/jira/browse/OOZIE-2538
 Project: Oozie
  Issue Type: Bug
  Components: core
Reporter: abhishek bafna
Assignee: abhishek bafna


We learned that

https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-5262 : 
http/conn/ssl/SSLConnectionSocketFactory.java in Apache HttpComponents 
HttpClient before 4.3.6 ignores the http.socket.timeout configuration setting 
during an SSL handshake, which allows remote attackers to cause a denial of 
service (HTTPS call hang) via unspecified vectors.

Also, Commons HttpClient project is now end of life, and is no longer being 
developed. It has been replaced by the Apache HttpComponents project in its 
HttpClient and HttpCore modules, which offer better performance and more 
flexibility.  http://hc.apache.org/httpclient-3.x/

Hence, HttpClient version should be updated.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2482) Pyspark job fails with Oozie

[Satish Subhashrao Saley (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15301134#comment-15301134
 ] 

Satish Subhashrao Saley commented on OOZIE-2482:


Hi [~rkanter] and [~gezapeti],
Setting {{SPARK_HOME}} approach is nice and saves us from passing zip files 
through --py-files option. 
But I am worried about [copying over the zip files again 
|https://github.com/apache/spark/blob/master/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L323-L327]
 by spark since they reside on local file system. Also we are copying them over 
from lib/ to python/lib/.
If we mention those using --py-files option with hdfs paths to pyspark.zip and 
py4j-0.9-src.zip then this copy could be avoided (i.e.if these files are part 
of sharelib, we can pass in that path). And for local mode, we just use the 
files which are available under local directory of launcher job. 

I think its between -- simpler code with extra copying Vs some involved code to 
avoid copying. 

with SPARK_HOME in yarn-cluster
{code}
2016-05-25 15:44:22,873 INFO [uber-SubtaskRunner] 
org.apache.spark.deploy.yarn.Client: Uploading resource 
file:/private/tmp/hadoop-saley/nm-local-dir/usercache/saley/appcache/application_1464215511846_0003/container_1464215511846_0003_01_01/python/lib/pyspark.zip
 -> 
hdfs://localhost:8020/user/saley/.sparkStaging/application_1464215511846_0004/pyspark.zip
2016-05-25 15:44:22,880 INFO [uber-SubtaskRunner] 
org.apache.spark.deploy.yarn.Client: Uploading resource 
file:/private/tmp/hadoop-saley/nm-local-dir/usercache/saley/appcache/application_1464215511846_0003/container_1464215511846_0003_01_01/python/lib/py4j-0.9-src.zip
 -> 
hdfs://localhost:8020/user/saley/.sparkStaging/application_1464215511846_0004/py4j-0.9-src.zip

{code}

with PYSPARK_ARCHIVES_PATH in yarn-cluster mode. I have setup zip files inside 
sharelib:
{code}
2016-05-25 23:35:43,440 INFO [uber-SubtaskRunner] 
org.apache.spark.deploy.yarn.Client: Source and destination file systems are 
the same. Not copying 
hdfs:/tmp/sharelib_dir/spark_yarn/share/spark/python/lib/pyspark.zip

2016-05-25 23:35:43,460 INFO [uber-SubtaskRunner] 
org.apache.spark.deploy.yarn.Client: Source and destination file systems are 
the same. Not copying 
hdfs:/tmp/sharelib_dir/spark_yarn/share/spark/python/lib/py4j-0.9-src.zip

{code}

> Pyspark job fails with Oozie
> 
>
> Key: OOZIE-2482
> URL: https://issues.apache.org/jira/browse/OOZIE-2482
> Project: Oozie
>  Issue Type: Bug
>  Components: core, workflow
>Affects Versions: 4.2.0
> Environment: Hadoop 2.7.2, Spark 1.6.0 on Yarn, Oozie 4.2.0
> Cluster secured with Kerberos
>Reporter: Alexandre Linte
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2482-1.patch, OOZIE-2482-2.patch, 
> OOZIE-2482-3.patch, OOZIE-2482-4.patch, OOZIE-2482-5.patch, 
> OOZIE-2482-6.patch, OOZIE-2482-zip.patch, py4j-0.9-src.zip, pyspark.zip
>
>
> Hello,
> I'm trying to run pi.py example in a pyspark job with Oozie. Every try I made 
> failed for the same reason: key not found: SPARK_HOME.
> Note: A scala job works well in the environment with Oozie.
> The logs on the executors are:
> {noformat}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/mnt/hd4/hadoop/yarn/local/filecache/145/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/mnt/hd2/hadoop/yarn/local/filecache/155/spark-assembly-1.6.0-hadoop2.7.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/opt/application/Hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException: 
> /mnt/hd7/hadoop/yarn/log/application_1454673025841_13136/container_1454673025841_13136_01_01
>  (Is a directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:142)
> at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
> at 
> org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
> at 
> org.apache.hadoop.yarn.ContainerLogAppender.activateOptions(ContainerLogAppender.java:55)
> at 
> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
> at 
> 

Build failed in Jenkins: oozie-trunk-precommit-build #2905

[Apache Jenkins Server]

See 

--
[...truncated 8398 lines...]
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-tools ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.3.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-mini ---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-mini ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-mini ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-mini ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-mini ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-mini ---
[INFO] Reading assembly descriptor: src/main/assemblies/empty.xml
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Distro 4.3.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-distro 
---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-distro 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-distro ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-distro ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-distro ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-distro ---
[INFO] 
[INFO] --- maven-antrun-plugin:1.6:run (default) @ oozie-distro ---
[INFO] Executing tasks

main:
  [get] Getting: 

[jira] [Commented] (OOZIE-2537) SqoopMain does not set up log4j properly

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15301054#comment-15301054
 ] 

Hadoop QA commented on OOZIE-2537:
--

Testing JIRA OOZIE-2537

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1780
.Tests failed: 4
.Tests errors: 0

.The patch failed the following testcases:

.  testJobLog(org.apache.oozie.client.TestOozieCLI)
.  testIDGeneration(org.apache.oozie.service.TestZKUUIDService)
.  
testbulkWfKillSuccess(org.apache.oozie.command.wf.TestBulkWorkflowXCommand)
.  testConnectionDrop(org.apache.oozie.jms.TestJMSJobEventListener)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/2905/

> SqoopMain does not set up log4j properly
> 
>
> Key: OOZIE-2537
> URL: https://issues.apache.org/jira/browse/OOZIE-2537
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: OOZIE-2537-001.patch, OOZIE-2537-002.patch
>
>
> SqoopMain does not set up log4j config properly for Sqoop. 
> It configures the "org.apache.hadoop.mapred" and 
> "org.apache.hadoop.mapreduce.Job" logger to use the "jobid" appender, but not 
> the "A" (console) appender, so MapReduce output is not visible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2482) Pyspark job fails with Oozie

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15300990#comment-15300990
 ] 

Hadoop QA commented on OOZIE-2482:
--

Testing JIRA OOZIE-2482

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 1 line(s) longer than 132 
characters
.{color:green}+1{color} the patch does adds/modifies 3 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1781
.Tests failed: 2
.Tests errors: 0

.The patch failed the following testcases:

.  testIDGeneration(org.apache.oozie.service.TestZKUUIDService)
.  testMultipleIDGeneration(org.apache.oozie.service.TestZKUUIDService)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/2904/

> Pyspark job fails with Oozie
> 
>
> Key: OOZIE-2482
> URL: https://issues.apache.org/jira/browse/OOZIE-2482
> Project: Oozie
>  Issue Type: Bug
>  Components: core, workflow
>Affects Versions: 4.2.0
> Environment: Hadoop 2.7.2, Spark 1.6.0 on Yarn, Oozie 4.2.0
> Cluster secured with Kerberos
>Reporter: Alexandre Linte
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2482-1.patch, OOZIE-2482-2.patch, 
> OOZIE-2482-3.patch, OOZIE-2482-4.patch, OOZIE-2482-5.patch, 
> OOZIE-2482-6.patch, OOZIE-2482-zip.patch, py4j-0.9-src.zip, pyspark.zip
>
>
> Hello,
> I'm trying to run pi.py example in a pyspark job with Oozie. Every try I made 
> failed for the same reason: key not found: SPARK_HOME.
> Note: A scala job works well in the environment with Oozie.
> The logs on the executors are:
> {noformat}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/mnt/hd4/hadoop/yarn/local/filecache/145/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/mnt/hd2/hadoop/yarn/local/filecache/155/spark-assembly-1.6.0-hadoop2.7.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/opt/application/Hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException: 
> /mnt/hd7/hadoop/yarn/log/application_1454673025841_13136/container_1454673025841_13136_01_01
>  (Is a directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:142)
> at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
> at 
> org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
> at 
> org.apache.hadoop.yarn.ContainerLogAppender.activateOptions(ContainerLogAppender.java:55)
> at 
> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
> at 
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
> at 
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
> at 
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
> at 
> 

Build failed in Jenkins: oozie-trunk-precommit-build #2904

[Apache Jenkins Server]

See 

--
[...truncated 26102 lines...]
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-tools ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.3.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-mini ---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-mini ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-mini ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-mini ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-mini ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-mini ---
[INFO] Reading assembly descriptor: src/main/assemblies/empty.xml
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Distro 4.3.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-distro 
---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-distro 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-distro ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-distro ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-distro ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-distro ---
[INFO] 
[INFO] --- maven-antrun-plugin:1.6:run 

[jira] [Commented] (OOZIE-2537) SqoopMain does not set up log4j properly

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15300905#comment-15300905
 ] 

Robert Kanter commented on OOZIE-2537:
--

+1 pending Jenkins.

> SqoopMain does not set up log4j properly
> 
>
> Key: OOZIE-2537
> URL: https://issues.apache.org/jira/browse/OOZIE-2537
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: OOZIE-2537-001.patch, OOZIE-2537-002.patch
>
>
> SqoopMain does not set up log4j config properly for Sqoop. 
> It configures the "org.apache.hadoop.mapred" and 
> "org.apache.hadoop.mapreduce.Job" logger to use the "jobid" appender, but not 
> the "A" (console) appender, so MapReduce output is not visible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2537) SqoopMain does not set up log4j properly

[Peter Bacsko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Bacsko updated OOZIE-2537:

Attachment: OOZIE-2537-002.patch

> SqoopMain does not set up log4j properly
> 
>
> Key: OOZIE-2537
> URL: https://issues.apache.org/jira/browse/OOZIE-2537
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: OOZIE-2537-001.patch, OOZIE-2537-002.patch
>
>
> SqoopMain does not set up log4j config properly for Sqoop. 
> It configures the "org.apache.hadoop.mapred" and 
> "org.apache.hadoop.mapreduce.Job" logger to use the "jobid" appender, but not 
> the "A" (console) appender, so MapReduce output is not visible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2537) SqoopMain does not set up log4j properly

[Peter Bacsko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Bacsko updated OOZIE-2537:

Description: 
SqoopMain does not set up log4j config properly for Sqoop. 

It configures the "org.apache.hadoop.mapred" and 
"org.apache.hadoop.mapreduce.Job" logger to use the "jobid" appender, but not 
the "A" (console) appender, so MapReduce output is not visible.

> SqoopMain does not set up log4j properly
> 
>
> Key: OOZIE-2537
> URL: https://issues.apache.org/jira/browse/OOZIE-2537
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>
> SqoopMain does not set up log4j config properly for Sqoop. 
> It configures the "org.apache.hadoop.mapred" and 
> "org.apache.hadoop.mapreduce.Job" logger to use the "jobid" appender, but not 
> the "A" (console) appender, so MapReduce output is not visible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (OOZIE-2482) Pyspark job fails with Oozie

[Peter Cseh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15300765#comment-15300765
 ] 

Peter Cseh edited comment on OOZIE-2482 at 5/25/16 8:10 PM:


Fixed up the documentation to explain that py files should be referred locally. 
PySpark does not like full hdfs paths in all running modes. After changing the 
path, the pi.py run in all 3 modes.
Added the dependency also. 


was (Author: gezapeti):
Fixed up the documentation to explain that py files should be referred locally. 
PySpark does not like full hdfs paths in all running modes. 

> Pyspark job fails with Oozie
> 
>
> Key: OOZIE-2482
> URL: https://issues.apache.org/jira/browse/OOZIE-2482
> Project: Oozie
>  Issue Type: Bug
>  Components: core, workflow
>Affects Versions: 4.2.0
> Environment: Hadoop 2.7.2, Spark 1.6.0 on Yarn, Oozie 4.2.0
> Cluster secured with Kerberos
>Reporter: Alexandre Linte
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2482-1.patch, OOZIE-2482-2.patch, 
> OOZIE-2482-3.patch, OOZIE-2482-4.patch, OOZIE-2482-5.patch, 
> OOZIE-2482-6.patch, OOZIE-2482-zip.patch, py4j-0.9-src.zip, pyspark.zip
>
>
> Hello,
> I'm trying to run pi.py example in a pyspark job with Oozie. Every try I made 
> failed for the same reason: key not found: SPARK_HOME.
> Note: A scala job works well in the environment with Oozie.
> The logs on the executors are:
> {noformat}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/mnt/hd4/hadoop/yarn/local/filecache/145/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/mnt/hd2/hadoop/yarn/local/filecache/155/spark-assembly-1.6.0-hadoop2.7.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/opt/application/Hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException: 
> /mnt/hd7/hadoop/yarn/log/application_1454673025841_13136/container_1454673025841_13136_01_01
>  (Is a directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:142)
> at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
> at 
> org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
> at 
> org.apache.hadoop.yarn.ContainerLogAppender.activateOptions(ContainerLogAppender.java:55)
> at 
> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
> at 
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
> at 
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
> at 
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:547)
> at 
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:483)
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at 
> org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:64)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:285)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
> at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:275)
> at 
> org.apache.hadoop.service.AbstractService.(AbstractService.java:43)
> Using properties file: null
> Parsed arguments:
>   master  yarn-master
>   deployMode  cluster
>   executorMemory  null
>   executorCores   null
>   totalExecutorCores  null
>   propertiesFile  null
>   driverMemorynull
>   driverCores null
>   driverExtraClassPathnull
>   driverExtraLibraryPath  null
>   driverExtraJavaOptions  null
>   supervise   false
>   queue   null
>   numExecutorsnull
>   files   null
>   pyFiles 

[jira] [Commented] (OOZIE-2482) Pyspark job fails with Oozie

[Peter Cseh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15300765#comment-15300765
 ] 

Peter Cseh commented on OOZIE-2482:
---

Fixed up the documentation to explain that py files should be referred locally. 
PySpark does not like full hdfs paths in all running modes. 

> Pyspark job fails with Oozie
> 
>
> Key: OOZIE-2482
> URL: https://issues.apache.org/jira/browse/OOZIE-2482
> Project: Oozie
>  Issue Type: Bug
>  Components: core, workflow
>Affects Versions: 4.2.0
> Environment: Hadoop 2.7.2, Spark 1.6.0 on Yarn, Oozie 4.2.0
> Cluster secured with Kerberos
>Reporter: Alexandre Linte
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2482-1.patch, OOZIE-2482-2.patch, 
> OOZIE-2482-3.patch, OOZIE-2482-4.patch, OOZIE-2482-5.patch, 
> OOZIE-2482-6.patch, OOZIE-2482-zip.patch, py4j-0.9-src.zip, pyspark.zip
>
>
> Hello,
> I'm trying to run pi.py example in a pyspark job with Oozie. Every try I made 
> failed for the same reason: key not found: SPARK_HOME.
> Note: A scala job works well in the environment with Oozie.
> The logs on the executors are:
> {noformat}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/mnt/hd4/hadoop/yarn/local/filecache/145/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/mnt/hd2/hadoop/yarn/local/filecache/155/spark-assembly-1.6.0-hadoop2.7.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/opt/application/Hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException: 
> /mnt/hd7/hadoop/yarn/log/application_1454673025841_13136/container_1454673025841_13136_01_01
>  (Is a directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:142)
> at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
> at 
> org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
> at 
> org.apache.hadoop.yarn.ContainerLogAppender.activateOptions(ContainerLogAppender.java:55)
> at 
> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
> at 
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
> at 
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
> at 
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:547)
> at 
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:483)
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at 
> org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:64)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:285)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
> at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:275)
> at 
> org.apache.hadoop.service.AbstractService.(AbstractService.java:43)
> Using properties file: null
> Parsed arguments:
>   master  yarn-master
>   deployMode  cluster
>   executorMemory  null
>   executorCores   null
>   totalExecutorCores  null
>   propertiesFile  null
>   driverMemorynull
>   driverCores null
>   driverExtraClassPathnull
>   driverExtraLibraryPath  null
>   driverExtraJavaOptions  null
>   supervise   false
>   queue   null
>   numExecutorsnull
>   files   null
>   pyFiles null
>   archivesnull
>   mainClass   null
>   primaryResource 
> hdfs://hadoopsandbox/User/toto/WORK/Oozie/pyspark/lib/pi.py
>   namePysparkpi example
>   childArgs   [100]
>   jarsnull
>   packages 

[jira] [Assigned] (OOZIE-2482) Pyspark job fails with Oozie

[Peter Cseh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh reassigned OOZIE-2482:
-

Assignee: Satish Subhashrao Saley  (was: Peter Cseh)

> Pyspark job fails with Oozie
> 
>
> Key: OOZIE-2482
> URL: https://issues.apache.org/jira/browse/OOZIE-2482
> Project: Oozie
>  Issue Type: Bug
>  Components: core, workflow
>Affects Versions: 4.2.0
> Environment: Hadoop 2.7.2, Spark 1.6.0 on Yarn, Oozie 4.2.0
> Cluster secured with Kerberos
>Reporter: Alexandre Linte
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2482-1.patch, OOZIE-2482-2.patch, 
> OOZIE-2482-3.patch, OOZIE-2482-4.patch, OOZIE-2482-5.patch, 
> OOZIE-2482-6.patch, OOZIE-2482-zip.patch, py4j-0.9-src.zip, pyspark.zip
>
>
> Hello,
> I'm trying to run pi.py example in a pyspark job with Oozie. Every try I made 
> failed for the same reason: key not found: SPARK_HOME.
> Note: A scala job works well in the environment with Oozie.
> The logs on the executors are:
> {noformat}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/mnt/hd4/hadoop/yarn/local/filecache/145/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/mnt/hd2/hadoop/yarn/local/filecache/155/spark-assembly-1.6.0-hadoop2.7.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/opt/application/Hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException: 
> /mnt/hd7/hadoop/yarn/log/application_1454673025841_13136/container_1454673025841_13136_01_01
>  (Is a directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:142)
> at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
> at 
> org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
> at 
> org.apache.hadoop.yarn.ContainerLogAppender.activateOptions(ContainerLogAppender.java:55)
> at 
> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
> at 
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
> at 
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
> at 
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:547)
> at 
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:483)
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at 
> org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:64)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:285)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
> at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:275)
> at 
> org.apache.hadoop.service.AbstractService.(AbstractService.java:43)
> Using properties file: null
> Parsed arguments:
>   master  yarn-master
>   deployMode  cluster
>   executorMemory  null
>   executorCores   null
>   totalExecutorCores  null
>   propertiesFile  null
>   driverMemorynull
>   driverCores null
>   driverExtraClassPathnull
>   driverExtraLibraryPath  null
>   driverExtraJavaOptions  null
>   supervise   false
>   queue   null
>   numExecutorsnull
>   files   null
>   pyFiles null
>   archivesnull
>   mainClass   null
>   primaryResource 
> hdfs://hadoopsandbox/User/toto/WORK/Oozie/pyspark/lib/pi.py
>   namePysparkpi example
>   childArgs   [100]
>   jarsnull
>   packagesnull
>   packagesExclusions  null
>   repositoriesnull
>   verbose true
> Spark 

[jira] [Assigned] (OOZIE-2482) Pyspark job fails with Oozie

[Peter Cseh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh reassigned OOZIE-2482:
-

Assignee: Peter Cseh  (was: Satish Subhashrao Saley)

> Pyspark job fails with Oozie
> 
>
> Key: OOZIE-2482
> URL: https://issues.apache.org/jira/browse/OOZIE-2482
> Project: Oozie
>  Issue Type: Bug
>  Components: core, workflow
>Affects Versions: 4.2.0
> Environment: Hadoop 2.7.2, Spark 1.6.0 on Yarn, Oozie 4.2.0
> Cluster secured with Kerberos
>Reporter: Alexandre Linte
>Assignee: Peter Cseh
> Attachments: OOZIE-2482-1.patch, OOZIE-2482-2.patch, 
> OOZIE-2482-3.patch, OOZIE-2482-4.patch, OOZIE-2482-5.patch, 
> OOZIE-2482-6.patch, OOZIE-2482-zip.patch, py4j-0.9-src.zip, pyspark.zip
>
>
> Hello,
> I'm trying to run pi.py example in a pyspark job with Oozie. Every try I made 
> failed for the same reason: key not found: SPARK_HOME.
> Note: A scala job works well in the environment with Oozie.
> The logs on the executors are:
> {noformat}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/mnt/hd4/hadoop/yarn/local/filecache/145/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/mnt/hd2/hadoop/yarn/local/filecache/155/spark-assembly-1.6.0-hadoop2.7.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/opt/application/Hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException: 
> /mnt/hd7/hadoop/yarn/log/application_1454673025841_13136/container_1454673025841_13136_01_01
>  (Is a directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:142)
> at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
> at 
> org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
> at 
> org.apache.hadoop.yarn.ContainerLogAppender.activateOptions(ContainerLogAppender.java:55)
> at 
> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
> at 
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
> at 
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
> at 
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:547)
> at 
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:483)
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at 
> org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:64)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:285)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
> at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:275)
> at 
> org.apache.hadoop.service.AbstractService.(AbstractService.java:43)
> Using properties file: null
> Parsed arguments:
>   master  yarn-master
>   deployMode  cluster
>   executorMemory  null
>   executorCores   null
>   totalExecutorCores  null
>   propertiesFile  null
>   driverMemorynull
>   driverCores null
>   driverExtraClassPathnull
>   driverExtraLibraryPath  null
>   driverExtraJavaOptions  null
>   supervise   false
>   queue   null
>   numExecutorsnull
>   files   null
>   pyFiles null
>   archivesnull
>   mainClass   null
>   primaryResource 
> hdfs://hadoopsandbox/User/toto/WORK/Oozie/pyspark/lib/pi.py
>   namePysparkpi example
>   childArgs   [100]
>   jarsnull
>   packagesnull
>   packagesExclusions  null
>   repositoriesnull
>   verbose true
> Spark properties 

Build failed in Jenkins: oozie-trunk-precommit-build #2903

[Apache Jenkins Server]

See 

--
[...truncated 7743 lines...]
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-tools 
---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-tools ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.3.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-mini ---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-mini ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-mini ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-mini ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-mini ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-mini ---
[INFO] Reading assembly descriptor: src/main/assemblies/empty.xml
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Distro 4.3.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-distro 
---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-distro 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-distro ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-distro ---

[jira] [Commented] (OOZIE-2482) Pyspark job fails with Oozie

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15300706#comment-15300706
 ] 

Robert Kanter commented on OOZIE-2482:
--

One last thing on the -5 patch: A dependency on commons-io is added to 
{{SparkMain}}, so we should be adding it to the spark sharelib explicitly 
instead of relying on Spark to pull it in.

I tried running a PySpark job and a Spark job on a nonsecure and secure 
cluster, with and without the zip files, and everything seems to be having as 
expected.  However, it's not working with {{yarn-client}} mode, only 
{{yarn-cluster}.  :(

Also, [~satishsaley], what do you think of the latest patch (other than the 
{{yarn-client}} issue)?  The approach is roughly the same, but it's setting 
different env vars.

> Pyspark job fails with Oozie
> 
>
> Key: OOZIE-2482
> URL: https://issues.apache.org/jira/browse/OOZIE-2482
> Project: Oozie
>  Issue Type: Bug
>  Components: core, workflow
>Affects Versions: 4.2.0
> Environment: Hadoop 2.7.2, Spark 1.6.0 on Yarn, Oozie 4.2.0
> Cluster secured with Kerberos
>Reporter: Alexandre Linte
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2482-1.patch, OOZIE-2482-2.patch, 
> OOZIE-2482-3.patch, OOZIE-2482-4.patch, OOZIE-2482-5.patch, 
> OOZIE-2482-zip.patch, py4j-0.9-src.zip, pyspark.zip
>
>
> Hello,
> I'm trying to run pi.py example in a pyspark job with Oozie. Every try I made 
> failed for the same reason: key not found: SPARK_HOME.
> Note: A scala job works well in the environment with Oozie.
> The logs on the executors are:
> {noformat}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/mnt/hd4/hadoop/yarn/local/filecache/145/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/mnt/hd2/hadoop/yarn/local/filecache/155/spark-assembly-1.6.0-hadoop2.7.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/opt/application/Hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException: 
> /mnt/hd7/hadoop/yarn/log/application_1454673025841_13136/container_1454673025841_13136_01_01
>  (Is a directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:142)
> at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
> at 
> org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
> at 
> org.apache.hadoop.yarn.ContainerLogAppender.activateOptions(ContainerLogAppender.java:55)
> at 
> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
> at 
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
> at 
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
> at 
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:547)
> at 
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:483)
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at 
> org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:64)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:285)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
> at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:275)
> at 
> org.apache.hadoop.service.AbstractService.(AbstractService.java:43)
> Using properties file: null
> Parsed arguments:
>   master  yarn-master
>   deployMode  cluster
>   executorMemory  null
>   executorCores   null
>   totalExecutorCores  null
>   propertiesFile  null
>   driverMemorynull
>   driverCores null
>   driverExtraClassPathnull
>   driverExtraLibraryPath  null
>   driverExtraJavaOptions  null
>   

[jira] [Commented] (OOZIE-2535) User can't disable uber mode

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15300707#comment-15300707
 ] 

Hadoop QA commented on OOZIE-2535:
--

Testing JIRA OOZIE-2535

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/2903/

> User can't disable uber mode
> 
>
> Key: OOZIE-2535
> URL: https://issues.apache.org/jira/browse/OOZIE-2535
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-2535-V1.patch, OOZIE-2535-V2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2501) ZK reentrant lock doesn't work for few cases

[Purshotam Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Purshotam Shah updated OOZIE-2501:
--
Description: 
We will have an issue when oozie trying to acquire a lock and at the same time, 
some other thread is releasing the same lock .

acquireLock will wait for 5 sec to acquire the lock. It will bypass the 
synchronized block and get lockEntry from the hashmap.

While it waiting for 5 sec to acquire the lock, other thread releases the lock 
and may execute the release code which will remove  lockEntry from the map.

If some other command from same thread tries to acquire the lock, it will 
create a new InterProcessReadWriteLock object and use that for acquiring the 
lock. 


Logic for lock acquiring.
{code}
 public LockToken getWriteLock(String resource, long wait) throws 
InterruptedException {
InterProcessReadWriteLock lockEntry;
synchronized (zkLocks) {
if (zkLocks.containsKey(resource)) {
lockEntry = zkLocks.get(resource);
}
else {
lockEntry = new InterProcessReadWriteLock(zk.getClient(), 
LOCKS_NODE + "/" + resource);
zkLocks.put(resource, lockEntry);
}
}
InterProcessMutex writeLock = lockEntry.writeLock();
return acquireLock(wait, writeLock, resource);
}
{code}
Logic for lock releasing

{code}
public void release() {
try {
lock.release();
if (zkLocks.get(resource) == null) {
return;
}
if (!isLockHeld()) {
synchronized (zkLocks) {
if (zkLocks.get(resource) != null) {
if (!isLockHeld()) {
zkLocks.remove(resource);
}
}
}
}
}
catch (Exception ex) {
LOG.warn("Could not release lock: " + ex.getMessage(), ex);
}

}
{code}

Curator code to acquire lock.

{code}
private boolean internalLock(long time, TimeUnit unit) throws Exception
{
/*
   Note on concurrency: a given lockData instance
   can be only acted on by a single thread so locking isn't necessary
*/

Thread  currentThread = Thread.currentThread();

LockDatalockData = threadData.get(currentThread);
if ( lockData != null )
{
// re-entering
lockData.lockCount.incrementAndGet();
return true;
}

String lockPath = internals.attemptLock(time, unit, getLockNodeBytes());
if ( lockPath != null )
{
LockDatanewLockData = new LockData(currentThread, lockPath);
threadData.put(currentThread, newLockData);
return true;
}

return false;
}
{code}


The approach we have followed is to use map with weakvalue. Once the lock is 
unreachable. GC will remove it from the map. We don't have to explicitly remove 
it.

  was:
We will have an issue when oozie trying to acquire a lock and at the same time, 
some other thread is releasing the same lock .

acquireLock will wait for 5 sec to acquire the lock. It will bypass the 
synchronized block and get lockEntry from the hashmap.

While it waiting for 5 sec to acquire the lock, other thread releases the lock 
and may execute the release code which will remove  lockEntry from the map.

If some other command from same thread tries to acquire the lock, it will 
create a new InterProcessReadWriteLock object and use that for acquiring the 
lock. 


Logic for lock acquiring.
{code}
 public LockToken getWriteLock(String resource, long wait) throws 
InterruptedException {
InterProcessReadWriteLock lockEntry;
synchronized (zkLocks) {
if (zkLocks.containsKey(resource)) {
lockEntry = zkLocks.get(resource);
}
else {
lockEntry = new InterProcessReadWriteLock(zk.getClient(), 
LOCKS_NODE + "/" + resource);
zkLocks.put(resource, lockEntry);
}
}
InterProcessMutex writeLock = lockEntry.writeLock();
return acquireLock(wait, writeLock, resource);
}
{code}
Logic for lock releasing

{code}
public void release() {
try {
lock.release();
if (zkLocks.get(resource) == null) {
return;
}
if (!isLockHeld()) {
synchronized (zkLocks) {
if (zkLocks.get(resource) != null) {
if (!isLockHeld()) {
zkLocks.remove(resource);
}
}
}
}
}

[jira] [Commented] (OOZIE-2509) SLA job status can stuck in running state

[Purshotam Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15300614#comment-15300614
 ] 

Purshotam Shah commented on OOZIE-2509:
---

Thanks Rohini for review. Committed to trunk.

> SLA job status can stuck in running state
> -
>
> Key: OOZIE-2509
> URL: https://issues.apache.org/jira/browse/OOZIE-2509
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-2509-V1.patch, OOZIE-2509-V2.patch, 
> OOZIE-2509-V3.patch, OOZIE-2509-V4.patch, OOZIE-2509-V5.patch, 
> OOZIE-2509-V6.patch, OOZIE-2509-V7.patch, OOZIE-2509-V8.patch
>
>
> There are few places where job status is not updated properly
> 1. Receiving event which is out of order. 
> Ex "oozie.service.EventHandlerService.batch.size"  is set to 50.
> oozie.service.EventHandlerService.worker.threads is set to 15. Which means 
> that there will be 15 thread processing event in the batch of 50.
> It can happen that 51th event gets process before the 49th event.
> If 49th  is job started event and 51th is job completed event, then the job 
> status will get overridden to running
> 2.
> {code}
> case COORDINATOR_ACTION:
> CoordinatorActionBean ca = jpaService.execute(new 
> CoordActionGetForSLAJPAExecutor(slaCalc.getId()));
> if (ca.isTerminalWithFailure()) {
> isEndMiss = ended = true;
> slaCalc.setActualEnd(ca.getLastModifiedTime());
> }
> if (ca.getExternalId() != null) {
> wf = jpaService.execute(new 
> WorkflowJobGetForSLAJPAExecutor(ca.getExternalId()));
> if (wf.getEndTime() != null) {
> ended = true;
> if (wf.getEndTime().getTime() > 
> slaCalc.getExpectedEnd().getTime()) {
> isEndMiss = true;
> }
> }
> slaCalc.setActualEnd(wf.getEndTime());
> slaCalc.setActualStart(wf.getStartTime());
> }
> {code}
> Oozie checks the wf status and update the sla status with coord job status.
> We might have a case where coord is still running,but wf has ended.
> 3. HistoryPurgeWorker updates endtime but doesn't update status.
> 4. There other few locking issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2535) User can't disable uber mode

[Purshotam Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Purshotam Shah updated OOZIE-2535:
--
Attachment: OOZIE-2535-V2.patch

> User can't disable uber mode
> 
>
> Key: OOZIE-2535
> URL: https://issues.apache.org/jira/browse/OOZIE-2535
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-2535-V1.patch, OOZIE-2535-V2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2533) Oozie Web UI gives Error 500 with Java 8u91

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15300477#comment-15300477
 ] 

Robert Kanter commented on OOZIE-2533:
--

We've done more testing and this seems like a problem with the Linux JDK 8u91 
(Mac works fine) with the current version of Oozie and even older ones.  This 
is really looking like a JDK bug.

> Oozie Web UI gives Error 500 with Java 8u91
> ---
>
> Key: OOZIE-2533
> URL: https://issues.apache.org/jira/browse/OOZIE-2533
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Robert Kanter
>
> When using Java 8u91, if I go to the Oozie Web UI in a browser, I get an 
> Error 500 with this:
> {noformat}
> org.apache.jasper.JasperException: Unable to compile class for JSP: 
> An error occurred at line: 1 in the generated java file
> The type java.io.ObjectInputStream cannot be resolved. It is indirectly 
> referenced from required .class files
> Stacktrace:
>   
> org.apache.jasper.compiler.DefaultErrorHandler.javacError(DefaultErrorHandler.java:92)
>   
> org.apache.jasper.compiler.ErrorDispatcher.javacError(ErrorDispatcher.java:330)
>   
> org.apache.jasper.compiler.JDTCompiler.generateClass(JDTCompiler.java:460)
>   org.apache.jasper.compiler.Compiler.compile(Compiler.java:356)
>   org.apache.jasper.compiler.Compiler.compile(Compiler.java:334)
>   org.apache.jasper.compiler.Compiler.compile(Compiler.java:321)
>   
> org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:592)
>   
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:328)
>   org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
>   org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
>   javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>   org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
>   
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:622)
>   
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:574)
>   org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
>   org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
> {noformat}
> The same info with a little more detail shows up in the localhost log file:
> {noformat}
> May 16, 2016 4:26:28 PM org.apache.catalina.core.StandardWrapperValve invoke
> SEVERE: Servlet.service() for servlet jsp threw exception
> org.apache.jasper.JasperException: Unable to compile class for JSP:
> An error occurred at line: 1 in the generated java file
> The type java.io.ObjectInputStream cannot be resolved. It is indirectly 
> referenced from required .class files
> Stacktrace:
> at 
> org.apache.jasper.compiler.DefaultErrorHandler.javacError(DefaultErrorHandler.java:92)
> at 
> org.apache.jasper.compiler.ErrorDispatcher.javacError(ErrorDispatcher.java:330)
> at 
> org.apache.jasper.compiler.JDTCompiler.generateClass(JDTCompiler.java:460)
> at org.apache.jasper.compiler.Compiler.compile(Compiler.java:356)
> at org.apache.jasper.compiler.Compiler.compile(Compiler.java:334)
> at org.apache.jasper.compiler.Compiler.compile(Compiler.java:321)
> at 
> org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:592)
> at 
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:328)
> at 
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
> at 
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:622)
> at 
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:574)
> at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at 
> org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

Oozie HA issue

[pavan kumar Kolamuri]

Hi all,

We have deployed oozie HA , we have seen issues like both servers are
trying to submit same workflow.

One workflow is failing with

Failing Oozie Launcher, Hadoop job Id mismatch, action file
[hdfs://cluster/user/pavan/oozie-oozi/0004867-160524102335554-oozie-oozi-W/registerpartitions--java/0004867-160524102335554-oozie-oozi-W@registerpartitions@0]
declares Id [job_1463560955979_320702] current Id [job_1463560955979_320739]


Other workflow is failing with

java.io.FileNotFoundException: File does not exist:
hdfs://cluster/user/pavan/oozie-oozi/0004867-160524102335554-oozie-oozi-W/registerpartitions--java/action.xml



Is it known bug ? Any help on this will be appreciated. Thanks in advance .


-- 
Regards
Pavan Kumar Kolamuri

[jira] [Commented] (OOZIE-2482) Pyspark job fails with Oozie

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15299883#comment-15299883
 ] 

Hadoop QA commented on OOZIE-2482:
--

Testing JIRA OOZIE-2482

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 1 line(s) longer than 132 
characters
.{color:green}+1{color} the patch does adds/modifies 3 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1781
.Tests failed: 3
.Tests errors: 0

.The patch failed the following testcases:

.  
testActionKillCommandDate(org.apache.oozie.command.coord.TestCoordActionsKillXCommand)
.  
testMemoryUsageAndSpeed(org.apache.oozie.service.TestPartitionDependencyManagerService)
.  
testBundleStatusTransitWithLock(org.apache.oozie.service.TestStatusTransitService)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/2902/

> Pyspark job fails with Oozie
> 
>
> Key: OOZIE-2482
> URL: https://issues.apache.org/jira/browse/OOZIE-2482
> Project: Oozie
>  Issue Type: Bug
>  Components: core, workflow
>Affects Versions: 4.2.0
> Environment: Hadoop 2.7.2, Spark 1.6.0 on Yarn, Oozie 4.2.0
> Cluster secured with Kerberos
>Reporter: Alexandre Linte
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2482-1.patch, OOZIE-2482-2.patch, 
> OOZIE-2482-3.patch, OOZIE-2482-4.patch, OOZIE-2482-5.patch, 
> OOZIE-2482-zip.patch, py4j-0.9-src.zip, pyspark.zip
>
>
> Hello,
> I'm trying to run pi.py example in a pyspark job with Oozie. Every try I made 
> failed for the same reason: key not found: SPARK_HOME.
> Note: A scala job works well in the environment with Oozie.
> The logs on the executors are:
> {noformat}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/mnt/hd4/hadoop/yarn/local/filecache/145/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/mnt/hd2/hadoop/yarn/local/filecache/155/spark-assembly-1.6.0-hadoop2.7.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/opt/application/Hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException: 
> /mnt/hd7/hadoop/yarn/log/application_1454673025841_13136/container_1454673025841_13136_01_01
>  (Is a directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:142)
> at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
> at 
> org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
> at 
> org.apache.hadoop.yarn.ContainerLogAppender.activateOptions(ContainerLogAppender.java:55)
> at 
> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
> at 
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
> at 
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
> at 
> 

Build failed in Jenkins: oozie-trunk-precommit-build #2902

[Apache Jenkins Server]

See 

--
[...truncated 26206 lines...]
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-tools ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.3.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-mini ---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-mini ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-mini ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-mini ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-mini ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-mini ---
[INFO] Reading assembly descriptor: src/main/assemblies/empty.xml
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Distro 4.3.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-distro 
---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-distro 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-distro ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-distro ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-distro ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-distro ---
[INFO] 
[INFO] --- maven-antrun-plugin:1.6:run 

Jenkins build is back to normal : oozie-trunk-find-patches-available #121390

[Apache Jenkins Server]

See 

Jenkins build is back to normal : oozie-trunk-find-patches-available #121388

[Apache Jenkins Server]

See 

[jira] [Comment Edited] (OOZIE-2482) Pyspark job fails with Oozie

[Peter Cseh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15299627#comment-15299627
 ] 

Peter Cseh edited comment on OOZIE-2482 at 5/25/16 8:00 AM:


Thank you for the review [~rkanter].
1) changed to:  These files can be added either to workflow's lib/ directory, 
to the sharelib or in sharelib mapping file.
2) link added with anchor 
3) duplications removed.
4) Fixed, just like in JavaActionExecutor.injectLauncherProperties(), we now 
add the original keys and the ones without oozie.launcher prefix. 
5) deleted
6) typo fixed
7) good idea, will do!



was (Author: gezapeti):
Thank you for the review [~rkanter].
1) changed to:  These files can be added either to workflow's lib/ directory, 
to the sharelib or in sharelib mapping file.
2) link added with anchor (I could not find a way to verify that the link 
works.)
3) duplications removed.
4) Fixed, just like in JavaActionExecutor.injectLauncherProperties(), we now 
add the original keys and the ones without oozie.launcher prefix. 
5) deleted
6) typo fixed
7) good idea, will do!


> Pyspark job fails with Oozie
> 
>
> Key: OOZIE-2482
> URL: https://issues.apache.org/jira/browse/OOZIE-2482
> Project: Oozie
>  Issue Type: Bug
>  Components: core, workflow
>Affects Versions: 4.2.0
> Environment: Hadoop 2.7.2, Spark 1.6.0 on Yarn, Oozie 4.2.0
> Cluster secured with Kerberos
>Reporter: Alexandre Linte
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2482-1.patch, OOZIE-2482-2.patch, 
> OOZIE-2482-3.patch, OOZIE-2482-4.patch, OOZIE-2482-5.patch, 
> OOZIE-2482-zip.patch, py4j-0.9-src.zip, pyspark.zip
>
>
> Hello,
> I'm trying to run pi.py example in a pyspark job with Oozie. Every try I made 
> failed for the same reason: key not found: SPARK_HOME.
> Note: A scala job works well in the environment with Oozie.
> The logs on the executors are:
> {noformat}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/mnt/hd4/hadoop/yarn/local/filecache/145/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/mnt/hd2/hadoop/yarn/local/filecache/155/spark-assembly-1.6.0-hadoop2.7.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/opt/application/Hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException: 
> /mnt/hd7/hadoop/yarn/log/application_1454673025841_13136/container_1454673025841_13136_01_01
>  (Is a directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:142)
> at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
> at 
> org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
> at 
> org.apache.hadoop.yarn.ContainerLogAppender.activateOptions(ContainerLogAppender.java:55)
> at 
> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
> at 
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
> at 
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
> at 
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:547)
> at 
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:483)
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at 
> org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:64)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:285)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
> at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:275)
> at 
> org.apache.hadoop.service.AbstractService.(AbstractService.java:43)
> Using properties file: null
> Parsed arguments:
>   master  yarn-master
>   

[jira] [Assigned] (OOZIE-2482) Pyspark job fails with Oozie

[Peter Cseh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh reassigned OOZIE-2482:
-

Assignee: Satish Subhashrao Saley  (was: Peter Cseh)

> Pyspark job fails with Oozie
> 
>
> Key: OOZIE-2482
> URL: https://issues.apache.org/jira/browse/OOZIE-2482
> Project: Oozie
>  Issue Type: Bug
>  Components: core, workflow
>Affects Versions: 4.2.0
> Environment: Hadoop 2.7.2, Spark 1.6.0 on Yarn, Oozie 4.2.0
> Cluster secured with Kerberos
>Reporter: Alexandre Linte
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2482-1.patch, OOZIE-2482-2.patch, 
> OOZIE-2482-3.patch, OOZIE-2482-4.patch, OOZIE-2482-5.patch, 
> OOZIE-2482-zip.patch, py4j-0.9-src.zip, pyspark.zip
>
>
> Hello,
> I'm trying to run pi.py example in a pyspark job with Oozie. Every try I made 
> failed for the same reason: key not found: SPARK_HOME.
> Note: A scala job works well in the environment with Oozie.
> The logs on the executors are:
> {noformat}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/mnt/hd4/hadoop/yarn/local/filecache/145/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/mnt/hd2/hadoop/yarn/local/filecache/155/spark-assembly-1.6.0-hadoop2.7.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/opt/application/Hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException: 
> /mnt/hd7/hadoop/yarn/log/application_1454673025841_13136/container_1454673025841_13136_01_01
>  (Is a directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:142)
> at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
> at 
> org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
> at 
> org.apache.hadoop.yarn.ContainerLogAppender.activateOptions(ContainerLogAppender.java:55)
> at 
> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
> at 
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
> at 
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
> at 
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:547)
> at 
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:483)
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at 
> org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:64)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:285)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
> at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:275)
> at 
> org.apache.hadoop.service.AbstractService.(AbstractService.java:43)
> Using properties file: null
> Parsed arguments:
>   master  yarn-master
>   deployMode  cluster
>   executorMemory  null
>   executorCores   null
>   totalExecutorCores  null
>   propertiesFile  null
>   driverMemorynull
>   driverCores null
>   driverExtraClassPathnull
>   driverExtraLibraryPath  null
>   driverExtraJavaOptions  null
>   supervise   false
>   queue   null
>   numExecutorsnull
>   files   null
>   pyFiles null
>   archivesnull
>   mainClass   null
>   primaryResource 
> hdfs://hadoopsandbox/User/toto/WORK/Oozie/pyspark/lib/pi.py
>   namePysparkpi example
>   childArgs   [100]
>   jarsnull
>   packagesnull
>   packagesExclusions  null
>   repositoriesnull
>   verbose true
> Spark properties used, 

[jira] [Updated] (OOZIE-2482) Pyspark job fails with Oozie

[Peter Cseh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh updated OOZIE-2482:
--
Attachment: OOZIE-2482-5.patch

> Pyspark job fails with Oozie
> 
>
> Key: OOZIE-2482
> URL: https://issues.apache.org/jira/browse/OOZIE-2482
> Project: Oozie
>  Issue Type: Bug
>  Components: core, workflow
>Affects Versions: 4.2.0
> Environment: Hadoop 2.7.2, Spark 1.6.0 on Yarn, Oozie 4.2.0
> Cluster secured with Kerberos
>Reporter: Alexandre Linte
>Assignee: Peter Cseh
> Attachments: OOZIE-2482-1.patch, OOZIE-2482-2.patch, 
> OOZIE-2482-3.patch, OOZIE-2482-4.patch, OOZIE-2482-5.patch, 
> OOZIE-2482-zip.patch, py4j-0.9-src.zip, pyspark.zip
>
>
> Hello,
> I'm trying to run pi.py example in a pyspark job with Oozie. Every try I made 
> failed for the same reason: key not found: SPARK_HOME.
> Note: A scala job works well in the environment with Oozie.
> The logs on the executors are:
> {noformat}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/mnt/hd4/hadoop/yarn/local/filecache/145/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/mnt/hd2/hadoop/yarn/local/filecache/155/spark-assembly-1.6.0-hadoop2.7.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/opt/application/Hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException: 
> /mnt/hd7/hadoop/yarn/log/application_1454673025841_13136/container_1454673025841_13136_01_01
>  (Is a directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:142)
> at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
> at 
> org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
> at 
> org.apache.hadoop.yarn.ContainerLogAppender.activateOptions(ContainerLogAppender.java:55)
> at 
> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
> at 
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
> at 
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
> at 
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:547)
> at 
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:483)
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at 
> org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:64)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:285)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
> at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:275)
> at 
> org.apache.hadoop.service.AbstractService.(AbstractService.java:43)
> Using properties file: null
> Parsed arguments:
>   master  yarn-master
>   deployMode  cluster
>   executorMemory  null
>   executorCores   null
>   totalExecutorCores  null
>   propertiesFile  null
>   driverMemorynull
>   driverCores null
>   driverExtraClassPathnull
>   driverExtraLibraryPath  null
>   driverExtraJavaOptions  null
>   supervise   false
>   queue   null
>   numExecutorsnull
>   files   null
>   pyFiles null
>   archivesnull
>   mainClass   null
>   primaryResource 
> hdfs://hadoopsandbox/User/toto/WORK/Oozie/pyspark/lib/pi.py
>   namePysparkpi example
>   childArgs   [100]
>   jarsnull
>   packagesnull
>   packagesExclusions  null
>   repositoriesnull
>   verbose true
> Spark properties used, including those specified through
>  --conf 

[jira] [Assigned] (OOZIE-2482) Pyspark job fails with Oozie

[Peter Cseh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh reassigned OOZIE-2482:
-

Assignee: Peter Cseh  (was: Satish Subhashrao Saley)

> Pyspark job fails with Oozie
> 
>
> Key: OOZIE-2482
> URL: https://issues.apache.org/jira/browse/OOZIE-2482
> Project: Oozie
>  Issue Type: Bug
>  Components: core, workflow
>Affects Versions: 4.2.0
> Environment: Hadoop 2.7.2, Spark 1.6.0 on Yarn, Oozie 4.2.0
> Cluster secured with Kerberos
>Reporter: Alexandre Linte
>Assignee: Peter Cseh
> Attachments: OOZIE-2482-1.patch, OOZIE-2482-2.patch, 
> OOZIE-2482-3.patch, OOZIE-2482-4.patch, OOZIE-2482-zip.patch, 
> py4j-0.9-src.zip, pyspark.zip
>
>
> Hello,
> I'm trying to run pi.py example in a pyspark job with Oozie. Every try I made 
> failed for the same reason: key not found: SPARK_HOME.
> Note: A scala job works well in the environment with Oozie.
> The logs on the executors are:
> {noformat}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/mnt/hd4/hadoop/yarn/local/filecache/145/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/mnt/hd2/hadoop/yarn/local/filecache/155/spark-assembly-1.6.0-hadoop2.7.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/opt/application/Hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException: 
> /mnt/hd7/hadoop/yarn/log/application_1454673025841_13136/container_1454673025841_13136_01_01
>  (Is a directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:142)
> at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
> at 
> org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
> at 
> org.apache.hadoop.yarn.ContainerLogAppender.activateOptions(ContainerLogAppender.java:55)
> at 
> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
> at 
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
> at 
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
> at 
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:547)
> at 
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:483)
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at 
> org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:64)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:285)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
> at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:275)
> at 
> org.apache.hadoop.service.AbstractService.(AbstractService.java:43)
> Using properties file: null
> Parsed arguments:
>   master  yarn-master
>   deployMode  cluster
>   executorMemory  null
>   executorCores   null
>   totalExecutorCores  null
>   propertiesFile  null
>   driverMemorynull
>   driverCores null
>   driverExtraClassPathnull
>   driverExtraLibraryPath  null
>   driverExtraJavaOptions  null
>   supervise   false
>   queue   null
>   numExecutorsnull
>   files   null
>   pyFiles null
>   archivesnull
>   mainClass   null
>   primaryResource 
> hdfs://hadoopsandbox/User/toto/WORK/Oozie/pyspark/lib/pi.py
>   namePysparkpi example
>   childArgs   [100]
>   jarsnull
>   packagesnull
>   packagesExclusions  null
>   repositoriesnull
>   verbose true
> Spark properties used, including those specified through
> 

[jira] [Assigned] (OOZIE-2482) Pyspark job fails with Oozie

[Peter Cseh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh reassigned OOZIE-2482:
-

Assignee: Satish Subhashrao Saley  (was: Peter Cseh)

> Pyspark job fails with Oozie
> 
>
> Key: OOZIE-2482
> URL: https://issues.apache.org/jira/browse/OOZIE-2482
> Project: Oozie
>  Issue Type: Bug
>  Components: core, workflow
>Affects Versions: 4.2.0
> Environment: Hadoop 2.7.2, Spark 1.6.0 on Yarn, Oozie 4.2.0
> Cluster secured with Kerberos
>Reporter: Alexandre Linte
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2482-1.patch, OOZIE-2482-2.patch, 
> OOZIE-2482-3.patch, OOZIE-2482-4.patch, OOZIE-2482-5.patch, 
> OOZIE-2482-zip.patch, py4j-0.9-src.zip, pyspark.zip
>
>
> Hello,
> I'm trying to run pi.py example in a pyspark job with Oozie. Every try I made 
> failed for the same reason: key not found: SPARK_HOME.
> Note: A scala job works well in the environment with Oozie.
> The logs on the executors are:
> {noformat}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/mnt/hd4/hadoop/yarn/local/filecache/145/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/mnt/hd2/hadoop/yarn/local/filecache/155/spark-assembly-1.6.0-hadoop2.7.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/opt/application/Hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException: 
> /mnt/hd7/hadoop/yarn/log/application_1454673025841_13136/container_1454673025841_13136_01_01
>  (Is a directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:142)
> at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
> at 
> org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
> at 
> org.apache.hadoop.yarn.ContainerLogAppender.activateOptions(ContainerLogAppender.java:55)
> at 
> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
> at 
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
> at 
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
> at 
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:547)
> at 
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:483)
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at 
> org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:64)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:285)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
> at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:275)
> at 
> org.apache.hadoop.service.AbstractService.(AbstractService.java:43)
> Using properties file: null
> Parsed arguments:
>   master  yarn-master
>   deployMode  cluster
>   executorMemory  null
>   executorCores   null
>   totalExecutorCores  null
>   propertiesFile  null
>   driverMemorynull
>   driverCores null
>   driverExtraClassPathnull
>   driverExtraLibraryPath  null
>   driverExtraJavaOptions  null
>   supervise   false
>   queue   null
>   numExecutorsnull
>   files   null
>   pyFiles null
>   archivesnull
>   mainClass   null
>   primaryResource 
> hdfs://hadoopsandbox/User/toto/WORK/Oozie/pyspark/lib/pi.py
>   namePysparkpi example
>   childArgs   [100]
>   jarsnull
>   packagesnull
>   packagesExclusions  null
>   repositoriesnull
>   verbose true
> Spark properties used, 

[jira] [Updated] (OOZIE-2482) Pyspark job fails with Oozie

[Peter Cseh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh updated OOZIE-2482:
--
Attachment: OOZIE-2482-5.patch

> Pyspark job fails with Oozie
> 
>
> Key: OOZIE-2482
> URL: https://issues.apache.org/jira/browse/OOZIE-2482
> Project: Oozie
>  Issue Type: Bug
>  Components: core, workflow
>Affects Versions: 4.2.0
> Environment: Hadoop 2.7.2, Spark 1.6.0 on Yarn, Oozie 4.2.0
> Cluster secured with Kerberos
>Reporter: Alexandre Linte
>Assignee: Peter Cseh
> Attachments: OOZIE-2482-1.patch, OOZIE-2482-2.patch, 
> OOZIE-2482-3.patch, OOZIE-2482-4.patch, OOZIE-2482-5.patch, 
> OOZIE-2482-zip.patch, py4j-0.9-src.zip, pyspark.zip
>
>
> Hello,
> I'm trying to run pi.py example in a pyspark job with Oozie. Every try I made 
> failed for the same reason: key not found: SPARK_HOME.
> Note: A scala job works well in the environment with Oozie.
> The logs on the executors are:
> {noformat}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/mnt/hd4/hadoop/yarn/local/filecache/145/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/mnt/hd2/hadoop/yarn/local/filecache/155/spark-assembly-1.6.0-hadoop2.7.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/opt/application/Hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException: 
> /mnt/hd7/hadoop/yarn/log/application_1454673025841_13136/container_1454673025841_13136_01_01
>  (Is a directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:142)
> at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
> at 
> org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
> at 
> org.apache.hadoop.yarn.ContainerLogAppender.activateOptions(ContainerLogAppender.java:55)
> at 
> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
> at 
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
> at 
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
> at 
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:547)
> at 
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:483)
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at 
> org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:64)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:285)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
> at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:275)
> at 
> org.apache.hadoop.service.AbstractService.(AbstractService.java:43)
> Using properties file: null
> Parsed arguments:
>   master  yarn-master
>   deployMode  cluster
>   executorMemory  null
>   executorCores   null
>   totalExecutorCores  null
>   propertiesFile  null
>   driverMemorynull
>   driverCores null
>   driverExtraClassPathnull
>   driverExtraLibraryPath  null
>   driverExtraJavaOptions  null
>   supervise   false
>   queue   null
>   numExecutorsnull
>   files   null
>   pyFiles null
>   archivesnull
>   mainClass   null
>   primaryResource 
> hdfs://hadoopsandbox/User/toto/WORK/Oozie/pyspark/lib/pi.py
>   namePysparkpi example
>   childArgs   [100]
>   jarsnull
>   packagesnull
>   packagesExclusions  null
>   repositoriesnull
>   verbose true
> Spark properties used, including those specified through
>  --conf 

[jira] [Updated] (OOZIE-2482) Pyspark job fails with Oozie

[Peter Cseh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh updated OOZIE-2482:
--
Attachment: (was: OOZIE-2482-5.patch)

> Pyspark job fails with Oozie
> 
>
> Key: OOZIE-2482
> URL: https://issues.apache.org/jira/browse/OOZIE-2482
> Project: Oozie
>  Issue Type: Bug
>  Components: core, workflow
>Affects Versions: 4.2.0
> Environment: Hadoop 2.7.2, Spark 1.6.0 on Yarn, Oozie 4.2.0
> Cluster secured with Kerberos
>Reporter: Alexandre Linte
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2482-1.patch, OOZIE-2482-2.patch, 
> OOZIE-2482-3.patch, OOZIE-2482-4.patch, OOZIE-2482-zip.patch, 
> py4j-0.9-src.zip, pyspark.zip
>
>
> Hello,
> I'm trying to run pi.py example in a pyspark job with Oozie. Every try I made 
> failed for the same reason: key not found: SPARK_HOME.
> Note: A scala job works well in the environment with Oozie.
> The logs on the executors are:
> {noformat}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/mnt/hd4/hadoop/yarn/local/filecache/145/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/mnt/hd2/hadoop/yarn/local/filecache/155/spark-assembly-1.6.0-hadoop2.7.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/opt/application/Hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException: 
> /mnt/hd7/hadoop/yarn/log/application_1454673025841_13136/container_1454673025841_13136_01_01
>  (Is a directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:142)
> at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
> at 
> org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
> at 
> org.apache.hadoop.yarn.ContainerLogAppender.activateOptions(ContainerLogAppender.java:55)
> at 
> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
> at 
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
> at 
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
> at 
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:547)
> at 
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:483)
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at 
> org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:64)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:285)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
> at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:275)
> at 
> org.apache.hadoop.service.AbstractService.(AbstractService.java:43)
> Using properties file: null
> Parsed arguments:
>   master  yarn-master
>   deployMode  cluster
>   executorMemory  null
>   executorCores   null
>   totalExecutorCores  null
>   propertiesFile  null
>   driverMemorynull
>   driverCores null
>   driverExtraClassPathnull
>   driverExtraLibraryPath  null
>   driverExtraJavaOptions  null
>   supervise   false
>   queue   null
>   numExecutorsnull
>   files   null
>   pyFiles null
>   archivesnull
>   mainClass   null
>   primaryResource 
> hdfs://hadoopsandbox/User/toto/WORK/Oozie/pyspark/lib/pi.py
>   namePysparkpi example
>   childArgs   [100]
>   jarsnull
>   packagesnull
>   packagesExclusions  null
>   repositoriesnull
>   verbose true
> Spark properties used, including those specified through
>  

[jira] [Commented] (OOZIE-2482) Pyspark job fails with Oozie

[Peter Cseh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15299627#comment-15299627
 ] 

Peter Cseh commented on OOZIE-2482:
---

Thank you for the review [~rkanter].
1) changed to:  These files can be added either to workflow's lib/ directory, 
to the sharelib or in sharelib mapping file.
2) link added with anchor (I could not find a way to verify that the link 
works.)
3) duplications removed.
4) Fixed, just like in JavaActionExecutor.injectLauncherProperties(), we now 
add the original keys and the ones with oozie.launcher removed. 
5) deleted
6) typo fixed
7) good idea, will do!


> Pyspark job fails with Oozie
> 
>
> Key: OOZIE-2482
> URL: https://issues.apache.org/jira/browse/OOZIE-2482
> Project: Oozie
>  Issue Type: Bug
>  Components: core, workflow
>Affects Versions: 4.2.0
> Environment: Hadoop 2.7.2, Spark 1.6.0 on Yarn, Oozie 4.2.0
> Cluster secured with Kerberos
>Reporter: Alexandre Linte
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2482-1.patch, OOZIE-2482-2.patch, 
> OOZIE-2482-3.patch, OOZIE-2482-4.patch, OOZIE-2482-zip.patch, 
> py4j-0.9-src.zip, pyspark.zip
>
>
> Hello,
> I'm trying to run pi.py example in a pyspark job with Oozie. Every try I made 
> failed for the same reason: key not found: SPARK_HOME.
> Note: A scala job works well in the environment with Oozie.
> The logs on the executors are:
> {noformat}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/mnt/hd4/hadoop/yarn/local/filecache/145/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/mnt/hd2/hadoop/yarn/local/filecache/155/spark-assembly-1.6.0-hadoop2.7.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/opt/application/Hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException: 
> /mnt/hd7/hadoop/yarn/log/application_1454673025841_13136/container_1454673025841_13136_01_01
>  (Is a directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:142)
> at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
> at 
> org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
> at 
> org.apache.hadoop.yarn.ContainerLogAppender.activateOptions(ContainerLogAppender.java:55)
> at 
> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
> at 
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
> at 
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
> at 
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:547)
> at 
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:483)
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at 
> org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:64)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:285)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
> at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:275)
> at 
> org.apache.hadoop.service.AbstractService.(AbstractService.java:43)
> Using properties file: null
> Parsed arguments:
>   master  yarn-master
>   deployMode  cluster
>   executorMemory  null
>   executorCores   null
>   totalExecutorCores  null
>   propertiesFile  null
>   driverMemorynull
>   driverCores null
>   driverExtraClassPathnull
>   driverExtraLibraryPath  null
>   driverExtraJavaOptions  null
>   supervise   false
>   queue   null
>   numExecutorsnull
>   files   null
>   pyFiles null
>   

[jira] [Comment Edited] (OOZIE-2482) Pyspark job fails with Oozie

[Peter Cseh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15299627#comment-15299627
 ] 

Peter Cseh edited comment on OOZIE-2482 at 5/25/16 7:54 AM:


Thank you for the review [~rkanter].
1) changed to:  These files can be added either to workflow's lib/ directory, 
to the sharelib or in sharelib mapping file.
2) link added with anchor (I could not find a way to verify that the link 
works.)
3) duplications removed.
4) Fixed, just like in JavaActionExecutor.injectLauncherProperties(), we now 
add the original keys and the ones without oozie.launcher prefix. 
5) deleted
6) typo fixed
7) good idea, will do!



was (Author: gezapeti):
Thank you for the review [~rkanter].
1) changed to:  These files can be added either to workflow's lib/ directory, 
to the sharelib or in sharelib mapping file.
2) link added with anchor (I could not find a way to verify that the link 
works.)
3) duplications removed.
4) Fixed, just like in JavaActionExecutor.injectLauncherProperties(), we now 
add the original keys and the ones with oozie.launcher removed. 
5) deleted
6) typo fixed
7) good idea, will do!


> Pyspark job fails with Oozie
> 
>
> Key: OOZIE-2482
> URL: https://issues.apache.org/jira/browse/OOZIE-2482
> Project: Oozie
>  Issue Type: Bug
>  Components: core, workflow
>Affects Versions: 4.2.0
> Environment: Hadoop 2.7.2, Spark 1.6.0 on Yarn, Oozie 4.2.0
> Cluster secured with Kerberos
>Reporter: Alexandre Linte
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2482-1.patch, OOZIE-2482-2.patch, 
> OOZIE-2482-3.patch, OOZIE-2482-4.patch, OOZIE-2482-zip.patch, 
> py4j-0.9-src.zip, pyspark.zip
>
>
> Hello,
> I'm trying to run pi.py example in a pyspark job with Oozie. Every try I made 
> failed for the same reason: key not found: SPARK_HOME.
> Note: A scala job works well in the environment with Oozie.
> The logs on the executors are:
> {noformat}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/mnt/hd4/hadoop/yarn/local/filecache/145/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/mnt/hd2/hadoop/yarn/local/filecache/155/spark-assembly-1.6.0-hadoop2.7.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/opt/application/Hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException: 
> /mnt/hd7/hadoop/yarn/log/application_1454673025841_13136/container_1454673025841_13136_01_01
>  (Is a directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:142)
> at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
> at 
> org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
> at 
> org.apache.hadoop.yarn.ContainerLogAppender.activateOptions(ContainerLogAppender.java:55)
> at 
> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
> at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
> at 
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
> at 
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
> at 
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502)
> at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:547)
> at 
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:483)
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at 
> org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:64)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:285)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
> at 
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
> at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:275)
> at 
> org.apache.hadoop.service.AbstractService.(AbstractService.java:43)
> Using properties file: null
> Parsed arguments:
>   master   

[jira] Subscription: Oozie Patch Available

[jira]

Issue Subscription
Filter: Oozie Patch Available (93 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-2535  User can't disable uber mode
https://issues.apache.org/jira/browse/OOZIE-2535
OOZIE-2534  Versioned action libs (similar to how SharedLibs works)
https://issues.apache.org/jira/browse/OOZIE-2534
OOZIE-2530  Remove jline from parent and sharelib pig pom
https://issues.apache.org/jira/browse/OOZIE-2530
OOZIE-2528  Print out environment variables in LauncherMapper
https://issues.apache.org/jira/browse/OOZIE-2528
OOZIE-2522  There can be multiple coord submit from bundle in case of ZK glitch
https://issues.apache.org/jira/browse/OOZIE-2522
OOZIE-2518  Oozie is creating many files and directories in /tmp per day
https://issues.apache.org/jira/browse/OOZIE-2518
OOZIE-2517  Add support for startCreatedTime and endCreatedTime filters for 
coord and bundles
https://issues.apache.org/jira/browse/OOZIE-2517
OOZIE-2516  Update web service documentation for jobs API
https://issues.apache.org/jira/browse/OOZIE-2516
OOZIE-2515  Duplicate information for "Changing endtime/pausetime of a Bundle 
Job" in CommandLineTool wiki
https://issues.apache.org/jira/browse/OOZIE-2515
OOZIE-2513  log.scan.duration should not be used for error and audit logs
https://issues.apache.org/jira/browse/OOZIE-2513
OOZIE-2512  ShareLibservice returns incorrect path for jar
https://issues.apache.org/jira/browse/OOZIE-2512
OOZIE-2509  SLA job status can stuck in running state
https://issues.apache.org/jira/browse/OOZIE-2509
OOZIE-2508  Documentation change for Coord action rerun [OOZIE-1735]
https://issues.apache.org/jira/browse/OOZIE-2508
OOZIE-2507  Expose monitoring via JMX beans in Oozie
https://issues.apache.org/jira/browse/OOZIE-2507
OOZIE-2506  Add logs into RecoverService for logging information about queued 
commnads
https://issues.apache.org/jira/browse/OOZIE-2506
OOZIE-2505  space on classpath causes error in spark-job
https://issues.apache.org/jira/browse/OOZIE-2505
OOZIE-2504  Create a log4j.properties under HADOOP_CONF_DIR in Shell Action
https://issues.apache.org/jira/browse/OOZIE-2504
OOZIE-2503  show ChildJobURLs to spark action
https://issues.apache.org/jira/browse/OOZIE-2503
OOZIE-2499  map-reduce launcher does not need distributed files, archives 
except  jar of input/outputformat class
https://issues.apache.org/jira/browse/OOZIE-2499
OOZIE-2498  Oozie CallerId configuration for downstream components 
https://issues.apache.org/jira/browse/OOZIE-2498
OOZIE-2497  Testcase fails on windows with hard coded URIs
https://issues.apache.org/jira/browse/OOZIE-2497
OOZIE-2496  Testcase failure due to environment specific line.separator value
https://issues.apache.org/jira/browse/OOZIE-2496
OOZIE-2495  change action status from  ErrorType.NON_TRANSIENT to TRANSIENT 
when SSH action occurs AUTH_FAILED occasionally
https://issues.apache.org/jira/browse/OOZIE-2495
OOZIE-2494  Cron syntax not handling DST properly
https://issues.apache.org/jira/browse/OOZIE-2494
OOZIE-2493  Remove action.xml file from local file system instead of hadoop 
file system
https://issues.apache.org/jira/browse/OOZIE-2493
OOZIE-2491  oozie acl cannot specify group,it does`t work
https://issues.apache.org/jira/browse/OOZIE-2491
OOZIE-2487  Unable to build distro on Java 8 (on Mac)
https://issues.apache.org/jira/browse/OOZIE-2487
OOZIE-2484  If coord missing dependencies are empty, Oozie uses new input-logic 
format to write data
https://issues.apache.org/jira/browse/OOZIE-2484
OOZIE-2482  Pyspark job fails with Oozie
https://issues.apache.org/jira/browse/OOZIE-2482
OOZIE-2476  When one of the action from fork fails with transient error, WF 
never joins.
https://issues.apache.org/jira/browse/OOZIE-2476
OOZIE-2475  Oozie does not cleanup failed actions; fills up namespace quota
https://issues.apache.org/jira/browse/OOZIE-2475
OOZIE-2473  Connection pool for SMTP connection
https://issues.apache.org/jira/browse/OOZIE-2473
OOZIE-2467  Oozie can shutdown itself on long GC pause
https://issues.apache.org/jira/browse/OOZIE-2467
OOZIE-2462  When calling ParamChecker.notNull() in 
CoordActionsIgnoreXCommand.java, "Action" should be passed instead of "Action 
cannot be null".
https://issues.apache.org/jira/browse/OOZIE-2462
OOZIE-2461  Workflow, Coordinator and Bundle job querying should have last 
modified filter
https://issues.apache.org/jira/browse/OOZIE-2461
OOZIE-2458  'oozie-setup.sh sharelib create' should ensure uploaded jars are 
world readable
https://issues.apache.org/jira/browse/OOZIE-2458
OOZIE-2457  Oozie log parsing regex