[jira] [Commented] (OOZIE-3440) Oozie Spark action replaces path symlink # to %23

2019-02-25 Thread Denes Bodo (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-3440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777664#comment-16777664
 ] 

Denes Bodo commented on OOZIE-3440:
---

Thanks [~asalamon74] for your comments, and thanks [~nobigo] for the testing. 
I've made the requested changes, please see the review board. Thanks.

> Oozie Spark action replaces path symlink # to %23
> -
>
> Key: OOZIE-3440
> URL: https://issues.apache.org/jira/browse/OOZIE-3440
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Affects Versions: 4.3.1, 5.1.0
> Environment: Hadoop 2.7.3
> Spark 2 - 2.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Major
> Attachments: OOZIE-3440-001.patch, OOZIE-3440-002.patch
>
>
> When we provide in a hive action:
>  * --files /etc/spark2/conf/hive-site.xml#hive-site.xml or
>  * /etc/spark2/conf/hive-site.xml#hive-site.xml or
> * hdfs:///tmp/spark2/conf/hive-site.xml#hive-site.xml or
>  * /etc/spark2/conf/hive-site.xml#hive-site.xml
> we get the following error:
> {code}
> Failing Oozie Launcher, Main class 
> [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, File 
> file:/etc/spark2/conf/hive-site.xml%23hive-site.xml does not exist
> {code}
> The culprit seems to be 
> https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java#L480L489
>  .
> Please help me confirm if this is a bug or not. Meanwhile I am creating a 
> fix/workaround to this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] Subscription: Oozie Patch Available

2019-02-25 Thread jira 
Issue Subscription
Filter: Oozie Patch Available (91 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3440  Oozie Spark action replaces path symlink # to %23
https://issues.apache.org/jira/browse/OOZIE-3440
OOZIE-3432  When use configuration oozie.service.ProxyUserService.root.hosts, 
oozie-error.log shows Invalid configuration defined
https://issues.apache.org/jira/browse/OOZIE-3432
OOZIE-3418  Upgrade to Guava 27
https://issues.apache.org/jira/browse/OOZIE-3418
OOZIE-3409  Oozie Server : Memory leak in EL evaluation
https://issues.apache.org/jira/browse/OOZIE-3409
OOZIE-3404  The env variable of SPARK_HOME needs to be set when running pySpark
https://issues.apache.org/jira/browse/OOZIE-3404
OOZIE-3375  Can't use empty  in coordinator
https://issues.apache.org/jira/browse/OOZIE-3375
OOZIE-3367  Using && in EL expressions in oozie bundle.xml files generates 
parse errors
https://issues.apache.org/jira/browse/OOZIE-3367
OOZIE-3366  Update workflow status and subworkflow status on suspend command
https://issues.apache.org/jira/browse/OOZIE-3366
OOZIE-3364  Rerunning Oozie bundle jobs starts the coordinators in 
indeterminate order
https://issues.apache.org/jira/browse/OOZIE-3364
OOZIE-3362  When killed, SSH action should kill the spawned processes on target 
host
https://issues.apache.org/jira/browse/OOZIE-3362
OOZIE-3320  Oozie ShellAction should support absolute bash file path
https://issues.apache.org/jira/browse/OOZIE-3320
OOZIE-3319  Log SSH action callback error output
https://issues.apache.org/jira/browse/OOZIE-3319
OOZIE-3301  Update NOTICE file
https://issues.apache.org/jira/browse/OOZIE-3301
OOZIE-3274  Remove slf4j
https://issues.apache.org/jira/browse/OOZIE-3274
OOZIE-3266  Coord action rerun support RERUN_SKIP_NODES option
https://issues.apache.org/jira/browse/OOZIE-3266
OOZIE-3265  properties RERUN_FAIL_NODES and RERUN_SKIP_NODES should be able to 
appear together
https://issues.apache.org/jira/browse/OOZIE-3265
OOZIE-3256  refactor OozieCLI class
https://issues.apache.org/jira/browse/OOZIE-3256
OOZIE-3249  [tools] Instrumentation log parser
https://issues.apache.org/jira/browse/OOZIE-3249
OOZIE-3199  Let system property restriction configurable
https://issues.apache.org/jira/browse/OOZIE-3199
OOZIE-3196  Authorization: restrict world readability by user
https://issues.apache.org/jira/browse/OOZIE-3196
OOZIE-3179  Adding a configurable config-default.xml location to a workflow
https://issues.apache.org/jira/browse/OOZIE-3179
OOZIE-3170  Oozie Diagnostic Bundle tool fails with NPE due to missing service 
class
https://issues.apache.org/jira/browse/OOZIE-3170
OOZIE-3137  Add support for log4j2 in HiveMain
https://issues.apache.org/jira/browse/OOZIE-3137
OOZIE-3135  Configure log4j2 in SqoopMain
https://issues.apache.org/jira/browse/OOZIE-3135
OOZIE-3091  Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: 
org/apache/avro/mapred/AvroWrapper"
https://issues.apache.org/jira/browse/OOZIE-3091
OOZIE-3071  Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 
than Spark 2.2.0
https://issues.apache.org/jira/browse/OOZIE-3071
OOZIE-3063  Sanitizing variables that are part of openjpa.ConnectionProperties
https://issues.apache.org/jira/browse/OOZIE-3063
OOZIE-3062  Set HADOOP_CONF_DIR for spark action
https://issues.apache.org/jira/browse/OOZIE-3062
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions
https://issues.apache.org/jira/browse/OOZIE-2812
OOZIE-2795  Create lib directory or symlink for Oozie CLI during packaging
https://issues.apache.org/jira/browse/OOZIE-2795
OOZIE-2784  Include WEEK as a parameter in the Coordinator Expression Language 
Evaulator
https://issues.apache.org/jira/browse/OOZIE-2784
OOZIE-2779  Mask Hive2 action Beeline JDBC password
https://issues.apache.org/jira/browse/OOZIE-2779
OOZIE-2736  Reduce the number of threads during test execution
https://issues.apache.org/jira/browse/OOZIE-2736
OOZIE-2714  Detect conflicting

[jira] [Commented] (OOZIE-3440) Oozie Spark action replaces path symlink # to %23

2019-02-25 Thread Hadoop QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-3440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776950#comment-16776950
 ] 

Hadoop QA commented on OOZIE-3440:
--


Testing JIRA OOZIE-3440

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warning(s)
.{color:orange}WARNING{color}: the current HEAD has 100 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/git].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in 
[fluent-job/fluent-job-api].
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [webapp].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [core].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 3140
.{color:orange}Tests failed at first run:{color}
TestPurgeXCommand#testPurgeableWorkflowPurgeableSubWorkflowPurgeableSubSubWorkflow
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/1019/



> Oozie Spark action replaces path symlink # to %23
> -
>
> Key: OOZIE-3440
> URL: https://issues.apache.org/jira/browse/OOZIE-3440
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Affects Versions: 4.3.1, 5.1.0
> Environment: Hadoop 2.7.3
> Spark 2 - 2.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Major
> Attachments: OOZIE-3440-001.patch, OOZIE-3440-002.patch
>
>
> When we provide in a hive action:
>  * --files /etc/spark2/conf/hive-site.xml#hive-site.xml or
>  * /etc/spark2/conf/hive-site.xml#hive-site.xml or
> * hdfs:///tmp/spark2/conf/hive-site.xml#hive-site.xml or
>  * /etc/spark2/conf/hive-site.xml#hive-site.xml
> we get the following error:
> {code}
> Failing Oozie Launcher, Main class 
> [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, File 
> file:/etc/spark2/conf/hive-site.xml%23hive-site.xml does not exist
> {code}
> The culprit seems to be 
> https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java#L480L489
>  .
> Please help me confirm if this is a bug or not. Meanwhile I am creating a 
>

Failed: OOZIE-3440 PreCommit Build #1019

2019-02-25 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-3440
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/1019/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 2.26 MB...]
[DEBUG] There are no new bugs found in [tools].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [fluent-job/fluent-job-api].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [server].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [webapp].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [examples].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [core].
[INFO] There are no new bugs found totally].
[TRACE] SpotBugs diffs checked and reports created
[TRACE] Summary file size is 2614 bytes
[TRACE] Full summary file size is 1525 bytes
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/SPOTBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
 removed
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/SPOTBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
 removed
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
  Running test-patch task DISTRO


Testing JIRA OOZIE-3440

Cleaning local git workspace



+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any star imports
+1 the patch does not introduce any line longer than 132
+1 the patch adds/modifies 1 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 Javadoc generation succeeded with the patch
+1 the patch does not seem to introduce new Javadoc warning(s)
WARNING: the current HEAD has 100 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
+1 There are no new bugs found in [sharelib/hive2].
+1 There are no new bugs found in [sharelib/spark].
+1 There are no new bugs found in [sharelib/oozie].
+1 There are no new bugs found in [sharelib/pig].
+1 There are no new bugs found in [sharelib/streaming].
+1 There are no new bugs found in [sharelib/hive].
+1 There are no new bugs found in [sharelib/distcp].
+1 There are no new bugs found in [sharelib/hcatalog].
+1 There are no new bugs found in [sharelib/sqoop].
+1 There are no new bugs found in [sharelib/git].
+1 There are no new bugs found in [client].
+1 There are no new bugs found in [docs].
+1 There are no new bugs found in [tools].
+1 There are no new bugs found in [fluent-job/fluent-job-api].
+1 There are no new bugs found in [server].
+1 There are no new bugs found in [webapp].
+1 There are no new bugs found in [examples].
+1 There are no new bugs found in [core].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 3140
Tests failed at first run:
TestPurgeXCommand#testPurgeableWorkflowPurgeableSubWorkflowPurgeableSubSubWorkflow
For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
+1 DISTRO
+1 distro tarball builds with the patch 


+1 Overall result, good!, no -1s

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/1019/

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:--

[jira] [Commented] (OOZIE-3206) The README.txt file refers to the 4.3.0 Quickstart docs

2019-02-25 Thread Andras Salamon (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-3206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776846#comment-16776846
 ] 

Andras Salamon commented on OOZIE-3206:
---

Probably we should create a symlink which points to the latest documentation, 
and we should use that in the README and in the documentations. After that we 
can just update that single symlink after a new release. Of course we should 
also add this information to the how [How to Release 
Oozie|https://cwiki.apache.org/confluence/display/OOZIE/How+To+Release] 
documentation.

> The README.txt file refers to the 4.3.0 Quickstart docs
> ---
>
> Key: OOZIE-3206
> URL: https://issues.apache.org/jira/browse/OOZIE-3206
> Project: Oozie
>  Issue Type: Bug
>  Components: docs
>Affects Versions: 5.0.0
>Reporter: Robert Kanter
>Priority: Trivial
>  Labels: newbie
> Fix For: 5.2.0
>
>
> The README.txt file refers to the 4.3.0 Quickstart docs.  It should refer to 
> whatever the current version we're on is.
> http://oozie.apache.org/docs/4.3.0/DG_QuickStart.html 
> It might be a good idea to update our website with a /docs/current/*.html 
> site so that we don't have to keep updating this, as long as we update the 
> "current".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3440) Oozie Spark action replaces path symlink # to %23

2019-02-25 Thread Hadoop QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-3440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776844#comment-16776844
 ] 

Hadoop QA commented on OOZIE-3440:
--

PreCommit-OOZIE-Build started


> Oozie Spark action replaces path symlink # to %23
> -
>
> Key: OOZIE-3440
> URL: https://issues.apache.org/jira/browse/OOZIE-3440
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Affects Versions: 4.3.1, 5.1.0
> Environment: Hadoop 2.7.3
> Spark 2 - 2.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Major
> Attachments: OOZIE-3440-001.patch, OOZIE-3440-002.patch
>
>
> When we provide in a hive action:
>  * --files /etc/spark2/conf/hive-site.xml#hive-site.xml or
>  * /etc/spark2/conf/hive-site.xml#hive-site.xml or
> * hdfs:///tmp/spark2/conf/hive-site.xml#hive-site.xml or
>  * /etc/spark2/conf/hive-site.xml#hive-site.xml
> we get the following error:
> {code}
> Failing Oozie Launcher, Main class 
> [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, File 
> file:/etc/spark2/conf/hive-site.xml%23hive-site.xml does not exist
> {code}
> The culprit seems to be 
> https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java#L480L489
>  .
> Please help me confirm if this is a bug or not. Meanwhile I am creating a 
> fix/workaround to this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 70042: Oozie Spark action replaces path symlink # to %23

2019-02-25 Thread Denes Bodo 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70042/
---

(Updated Feb. 25, 2019, 1:42 p.m.)


Review request for oozie.


Bugs: OOZIE-3440
https://issues.apache.org/jira/browse/OOZIE-3440


Repository: oozie-git


Description
---

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], 
main() threw exception, File 
file:/etc/spark2/conf/hive-site.xml%23hive-site.xml does not exist


Diffs (updated)
-

  
sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java
 28d9c5cc4 
  
sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkArgsExtractor.java
 7ccd26ad5 


Diff: https://reviews.apache.org/r/70042/diff/2/

Changes: https://reviews.apache.org/r/70042/diff/1-2/


Testing
---

Unit tests run


Thanks,

Denes Bodo

Re: Review Request 70042: Oozie Spark action replaces path symlink # to %23

2019-02-25 Thread Denes Bodo 


> On Feb. 25, 2019, 11:04 a.m., Andras Salamon wrote:
> > sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkArgsExtractor.java
> > Lines 398 (patched)
> > 
> >
> > There are two ways to specify the files:
> > 
> > --files 
> > --files=
> > 
> > Could you please also test the second type.

I think for this 4 cases creating parametrized tests does not worth.


> On Feb. 25, 2019, 11:04 a.m., Andras Salamon wrote:
> > sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkArgsExtractor.java
> > Lines 410-414 (patched)
> > 
> >
> > I think it would be possible to create better assertions here without 
> > converting the List to String. 
> > 
> > Could you please also add assert messages.

In an array list I cannot see a better check. Any idea is welcome.


- Denes


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70042/#review213151
---


On Feb. 25, 2019, 1:42 p.m., Denes Bodo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70042/
> ---
> 
> (Updated Feb. 25, 2019, 1:42 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-3440
> https://issues.apache.org/jira/browse/OOZIE-3440
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Failing Oozie Launcher, Main class 
> [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, File 
> file:/etc/spark2/conf/hive-site.xml%23hive-site.xml does not exist
> 
> 
> Diffs
> -
> 
>   
> sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java
>  28d9c5cc4 
>   
> sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkArgsExtractor.java
>  7ccd26ad5 
> 
> 
> Diff: https://reviews.apache.org/r/70042/diff/2/
> 
> 
> Testing
> ---
> 
> Unit tests run
> 
> 
> Thanks,
> 
> Denes Bodo
> 
>

[jira] [Updated] (OOZIE-3440) Oozie Spark action replaces path symlink # to %23

2019-02-25 Thread Denes Bodo (JIRA) 


 [ 
https://issues.apache.org/jira/browse/OOZIE-3440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denes Bodo updated OOZIE-3440:
--
Attachment: OOZIE-3440-002.patch

> Oozie Spark action replaces path symlink # to %23
> -
>
> Key: OOZIE-3440
> URL: https://issues.apache.org/jira/browse/OOZIE-3440
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Affects Versions: 4.3.1, 5.1.0
> Environment: Hadoop 2.7.3
> Spark 2 - 2.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Major
> Attachments: OOZIE-3440-001.patch, OOZIE-3440-002.patch
>
>
> When we provide in a hive action:
>  * --files /etc/spark2/conf/hive-site.xml#hive-site.xml or
>  * /etc/spark2/conf/hive-site.xml#hive-site.xml or
> * hdfs:///tmp/spark2/conf/hive-site.xml#hive-site.xml or
>  * /etc/spark2/conf/hive-site.xml#hive-site.xml
> we get the following error:
> {code}
> Failing Oozie Launcher, Main class 
> [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, File 
> file:/etc/spark2/conf/hive-site.xml%23hive-site.xml does not exist
> {code}
> The culprit seems to be 
> https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java#L480L489
>  .
> Please help me confirm if this is a bug or not. Meanwhile I am creating a 
> fix/workaround to this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3282) Prototyping: create Angular 6 workflow list page

2019-02-25 Thread Andras Salamon (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776776#comment-16776776
 ] 

Andras Salamon commented on OOZIE-3282:
---

In the meantime Angular 7 was released, so even if we go forward with angular, 
we should target that one.

> Prototyping: create Angular 6 workflow list page
> 
>
> Key: OOZIE-3282
> URL: https://issues.apache.org/jira/browse/OOZIE-3282
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Andras Piros
>Assignee: Mate Juhasz
>Priority: Major
>
> Create an [*Angular 6*|https://angular.io/docs] prototype of the workflow 
> list page. Don't have to handle authentication / authorization for now.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3441) Spark Action：Jackson version is too old 2.4.2

2019-02-25 Thread duan xiong (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-3441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776774#comment-16776774
 ] 

duan xiong commented on OOZIE-3441:
---

[~asalamon74].Hi,I have tried jackson 2.6.5 jackson version. It can still works 
 in Spark 1.6 and Spark 2.1. Only make jackson-module-scala*.jar version keep 
consistent with jackson. I will try to repair this.

> Spark Action：Jackson version is too old 2.4.2
> -
>
> Key: OOZIE-3441
> URL: https://issues.apache.org/jira/browse/OOZIE-3441
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Affects Versions: 5.1.0
>Reporter: duan xiong
>Assignee: duan xiong
>Priority: Critical
>
> When I install oozie 5.1.0, I run Spark Action Demo.It failed.In Yarn log,It 
> shows:
> {color:#FF}Caused By : 
> com.fasterxml.jackson.databind.JsonMappingException: Jackson version is too 
> old 2.4.2 Failing Oozie Launcher,null{color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3282) Prototyping: create Angular 6 workflow list page

2019-02-25 Thread Mate Juhasz (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776773#comment-16776773
 ] 

Mate Juhasz commented on OOZIE-3282:


[~kmarton] I didnt have time to work on it and I found the react protoype 
really great, maybe we should go with that one

> Prototyping: create Angular 6 workflow list page
> 
>
> Key: OOZIE-3282
> URL: https://issues.apache.org/jira/browse/OOZIE-3282
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Andras Piros
>Assignee: Mate Juhasz
>Priority: Major
>
> Create an [*Angular 6*|https://angular.io/docs] prototype of the workflow 
> list page. Don't have to handle authentication / authorization for now.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3440) Oozie Spark action replaces path symlink # to %23

2019-02-25 Thread Andras Salamon (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-3440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776707#comment-16776707
 ] 

Andras Salamon commented on OOZIE-3440:
---

[~nobigo] Thanks for the test

[~dionusos] I left a few comments on review board. Mostly test enhancements. 

> Oozie Spark action replaces path symlink # to %23
> -
>
> Key: OOZIE-3440
> URL: https://issues.apache.org/jira/browse/OOZIE-3440
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Affects Versions: 4.3.1, 5.1.0
> Environment: Hadoop 2.7.3
> Spark 2 - 2.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Major
> Attachments: OOZIE-3440-001.patch
>
>
> When we provide in a hive action:
>  * --files /etc/spark2/conf/hive-site.xml#hive-site.xml or
>  * /etc/spark2/conf/hive-site.xml#hive-site.xml or
> * hdfs:///tmp/spark2/conf/hive-site.xml#hive-site.xml or
>  * /etc/spark2/conf/hive-site.xml#hive-site.xml
> we get the following error:
> {code}
> Failing Oozie Launcher, Main class 
> [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, File 
> file:/etc/spark2/conf/hive-site.xml%23hive-site.xml does not exist
> {code}
> The culprit seems to be 
> https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java#L480L489
>  .
> Please help me confirm if this is a bug or not. Meanwhile I am creating a 
> fix/workaround to this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 70042: Oozie Spark action replaces path symlink # to %23

2019-02-25 Thread Andras Salamon 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70042/#review213151
---




sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java
Lines 491 (patched)


Please create a unit test for this method



sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java
Lines 492 (patched)


Null pointer check would be useful here.



sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java
Lines 493 (patched)


Please add a space after "for" and later after "if"



sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkArgsExtractor.java
Lines 397 (patched)


It seems to me you both modified --files and --archives options. Could you 
please also test --archives.



sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkArgsExtractor.java
Lines 398 (patched)


There are two ways to specify the files:

--files 
--files=

Could you please also test the second type.



sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkArgsExtractor.java
Lines 405 (patched)


It would be better to use different file names for the non-hdfs and hdfs 
version. Using the same name might result strange problems.



sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkArgsExtractor.java
Lines 410-414 (patched)


I think it would be possible to create better assertions here without 
converting the List to String. 

Could you please also add assert messages.


- Andras Salamon


On Feb. 22, 2019, 1:28 p.m., Denes Bodo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70042/
> ---
> 
> (Updated Feb. 22, 2019, 1:28 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-3440
> https://issues.apache.org/jira/browse/OOZIE-3440
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Failing Oozie Launcher, Main class 
> [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, File 
> file:/etc/spark2/conf/hive-site.xml%23hive-site.xml does not exist
> 
> 
> Diffs
> -
> 
>   
> sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java
>  28d9c5cc4 
>   
> sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkArgsExtractor.java
>  7ccd26ad5 
> 
> 
> Diff: https://reviews.apache.org/r/70042/diff/1/
> 
> 
> Testing
> ---
> 
> Unit tests run
> 
> 
> Thanks,
> 
> Denes Bodo
> 
>

[jira] [Created] (OOZIE-3443) Migrate from joda time to java.time

2019-02-25 Thread Andras Salamon (JIRA) 
Andras Salamon created OOZIE-3443:
-

 Summary: Migrate from joda time to java.time
 Key: OOZIE-3443
 URL: https://issues.apache.org/jira/browse/OOZIE-3443
 Project: Oozie
  Issue Type: Improvement
Affects Versions: trunk
Reporter: Andras Salamon


>From Java SE 8 onwards, users are asked to migrate to from [joda 
>time|https://www.joda.org/joda-time/] to java.time (JSR-310).

It seems to me, that we only use it directly in one place: 
[TestWorkflowActionRetryInfoXCommand.java|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionRetryInfoXCommand.java]
 so it would be not too difficult to migrate.





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (OOZIE-3442) Check and eliminate builds/runHudsonCIBuild.sh

2019-02-25 Thread duan xiong (JIRA) 


 [ 
https://issues.apache.org/jira/browse/OOZIE-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

duan xiong reassigned OOZIE-3442:
-

Assignee: duan xiong

> Check and eliminate builds/runHudsonCIBuild.sh
> --
>
> Key: OOZIE-3442
> URL: https://issues.apache.org/jira/browse/OOZIE-3442
> Project: Oozie
>  Issue Type: Improvement
>  Components: build
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: duan xiong
>Priority: Minor
>
> [runHudsonCIBuild.sh|https://github.com/apache/oozie/blob/master/builds/runHudsonCIBuild.sh]
>  is a rather old file, last time it was modified in 2002. It even skipped the 
> shell script modification of OOZIE-2352. The name of the script refers to 
> [Hudson|https://en.wikipedia.org/wiki/Hudson_(software)], which was long time 
> replaced by Jenkins.
> I don't think we use this script for anything, so probably it would be the 
> best to just delete it.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (OOZIE-3442) Check and eliminate builds/runHudsonCIBuild.sh

2019-02-25 Thread Andras Salamon (JIRA) 
Andras Salamon created OOZIE-3442:
-

 Summary: Check and eliminate builds/runHudsonCIBuild.sh
 Key: OOZIE-3442
 URL: https://issues.apache.org/jira/browse/OOZIE-3442
 Project: Oozie
  Issue Type: Improvement
  Components: build
Affects Versions: trunk
Reporter: Andras Salamon


[runHudsonCIBuild.sh|https://github.com/apache/oozie/blob/master/builds/runHudsonCIBuild.sh]
 is a rather old file, last time it was modified in 2002. It even skipped the 
shell script modification of OOZIE-2352. The name of the script refers to 
[Hudson|https://en.wikipedia.org/wiki/Hudson_(software)], which was long time 
replaced by Jenkins.

I don't think we use this script for anything, so probably it would be the best 
to just delete it.

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3441) Spark Action：Jackson version is too old 2.4.2

2019-02-25 Thread Andras Salamon (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-3441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776611#comment-16776611
 ] 

Andras Salamon commented on OOZIE-3441:
---

[~nobigo] OK thanks, just please be sure that it still works with Spark 1.6

> Spark Action：Jackson version is too old 2.4.2
> -
>
> Key: OOZIE-3441
> URL: https://issues.apache.org/jira/browse/OOZIE-3441
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Affects Versions: 5.1.0
>Reporter: duan xiong
>Assignee: duan xiong
>Priority: Critical
>
> When I install oozie 5.1.0, I run Spark Action Demo.It failed.In Yarn log,It 
> shows:
> {color:#FF}Caused By : 
> com.fasterxml.jackson.databind.JsonMappingException: Jackson version is too 
> old 2.4.2 Failing Oozie Launcher,null{color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)