[jira] [Updated] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail

2017-12-08 Thread Satish Subhashrao Saley (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-2787:
---

Cherry-picked from master to branch-4.3.


> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Fix For: 4.3.1
>
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch, 
> OOZIE-2787-3.patch, OOZIE-2787-4.patch, OOZIE-2787-5.patch, 
> OOZIE-2787-amend-1.patch, OOZIE-2787-amend-2.patch, OOZIE-2787-amend-3.patch, 
> OOZIE-2787-amend-4.patch, OOZIE-2787-amend-5.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to 
> distributed cache. Since this gets added twice, the job fails. This is 
> observed from spark 2.1.0 which introduces a check for same file and fails 
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue 
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail

2017-02-09 Thread Satish Subhashrao Saley (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-2787:
---
Attachment: OOZIE-2787-amend-5.patch

Renaming the title for section in documentation. 

> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch, 
> OOZIE-2787-3.patch, OOZIE-2787-4.patch, OOZIE-2787-5.patch, 
> OOZIE-2787-amend-1.patch, OOZIE-2787-amend-2.patch, OOZIE-2787-amend-3.patch, 
> OOZIE-2787-amend-4.patch, OOZIE-2787-amend-5.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to 
> distributed cache. Since this gets added twice, the job fails. This is 
> observed from spark 2.1.0 which introduces a check for same file and fails 
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue 
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail

2017-02-09 Thread Satish Subhashrao Saley (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-2787:
---
Attachment: OOZIE-2787-amend-4.patch

> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch, 
> OOZIE-2787-3.patch, OOZIE-2787-4.patch, OOZIE-2787-5.patch, 
> OOZIE-2787-amend-1.patch, OOZIE-2787-amend-2.patch, OOZIE-2787-amend-3.patch, 
> OOZIE-2787-amend-4.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to 
> distributed cache. Since this gets added twice, the job fails. This is 
> observed from spark 2.1.0 which introduces a check for same file and fails 
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue 
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail

2017-02-09 Thread Satish Subhashrao Saley (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-2787:
---
Attachment: OOZIE-2787-amend-3.patch

> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch, 
> OOZIE-2787-3.patch, OOZIE-2787-4.patch, OOZIE-2787-5.patch, 
> OOZIE-2787-amend-1.patch, OOZIE-2787-amend-2.patch, OOZIE-2787-amend-3.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to 
> distributed cache. Since this gets added twice, the job fails. This is 
> observed from spark 2.1.0 which introduces a check for same file and fails 
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue 
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail

2017-02-09 Thread Satish Subhashrao Saley (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-2787:
---
Attachment: OOZIE-2787-amend-2.patch

> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch, 
> OOZIE-2787-3.patch, OOZIE-2787-4.patch, OOZIE-2787-5.patch, 
> OOZIE-2787-amend-1.patch, OOZIE-2787-amend-2.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to 
> distributed cache. Since this gets added twice, the job fails. This is 
> observed from spark 2.1.0 which introduces a check for same file and fails 
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue 
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail

2017-02-08 Thread Satish Subhashrao Saley (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-2787:
---
Attachment: OOZIE-2787-amend-1.patch

> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch, 
> OOZIE-2787-3.patch, OOZIE-2787-4.patch, OOZIE-2787-5.patch, 
> OOZIE-2787-amend-1.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to 
> distributed cache. Since this gets added twice, the job fails. This is 
> observed from spark 2.1.0 which introduces a check for same file and fails 
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue 
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail

2017-02-03 Thread Satish Subhashrao Saley (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-2787:
---
Attachment: OOZIE-2787-5.patch

> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch, 
> OOZIE-2787-3.patch, OOZIE-2787-4.patch, OOZIE-2787-5.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to 
> distributed cache. Since this gets added twice, the job fails. This is 
> observed from spark 2.1.0 which introduces a check for same file and fails 
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue 
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail

2017-02-03 Thread Satish Subhashrao Saley (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-2787:
---
Attachment: OOZIE-2787-4.patch

- minor changes in java doc comment

> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch, 
> OOZIE-2787-3.patch, OOZIE-2787-4.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to 
> distributed cache. Since this gets added twice, the job fails. This is 
> observed from spark 2.1.0 which introduces a check for same file and fails 
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue 
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail

2017-02-03 Thread Satish Subhashrao Saley (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-2787:
---
Attachment: OOZIE-2787-3.patch

- Made class static
- Made listUri private

> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch, 
> OOZIE-2787-3.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to 
> distributed cache. Since this gets added twice, the job fails. This is 
> observed from spark 2.1.0 which introduces a check for same file and fails 
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue 
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail

2017-02-03 Thread Satish Subhashrao Saley (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-2787:
---
Attachment: OOZIE-2787-2.patch

- OOZIE-2554 is created for adding test cases for Spark action.
- Upgraded spark version in spark-2 profile
- updated filterJars method and included in separate class. OOZIE-2554 would 
add test cases for it.


> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to 
> distributed cache. Since this gets added twice, the job fails. This is 
> observed from spark 2.1.0 which introduces a check for same file and fails 
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue 
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)