[jira] [Updated] (OOZIE-2718) Improve -dryrun for bundles

2017-02-24 Thread Xiaobin Zheng (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobin Zheng updated OOZIE-2718:
-
Attachment: OOZIE-2718-6.patch

Thx for reviewing this [~qwertymaniac]!

I updated the documentation message according to your suggestion in patch 6.
[~abhishekbafna] [~rkanter] Can you please help review the latest patch?

> Improve -dryrun for bundles
> ---
>
> Key: OOZIE-2718
> URL: https://issues.apache.org/jira/browse/OOZIE-2718
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Xiaobin Zheng
> Attachments: OOZIE-2718-1.patch, OOZIE-2718-2.patch, 
> OOZIE-2718-3.patch, OOZIE-2718-4.patch, OOZIE-2718-5.patch, OOZIE-2718-6.patch
>
>
> When trying to do a -dryrun for Oozie:
> {code}
> error:
> Error: E1310 : E1310: Bundle Job submission Error: [null]
> {code}
> The 
> [documentation|https://oozie.apache.org/docs/4.2.0/DG_CommandLineTool.html#Dryrun_of_Coordinator_Job]
>  states that dryruns are working for workflows and coordinators.
> {quote}
> "Dryrun a workflow (since 3.3.2) or coordinator (since 2.0) job without 
> actually executing it"
> {quote}
> The easy fix is to document that it does not work and create an appropriate 
> error message for it. It's also possible to add support for bundle dryruns. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2796) oozie.action.keep.action.dir not getting noticed

2017-02-24 Thread Xiaobin Zheng (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobin Zheng updated OOZIE-2796:
-
Attachment: OOZIE-2796-4.patch

Thanks for the comments [~rkanter]! 

I am attaching patch 4:
1. created a constant in WorkflowXCommand.java
2. made the exception handling unit test case more specific by asserting the 
error code.
Regarding 3, each action tries to clean up the action dir, and upon workflow 
end we clean up workflow dir. There seems very subtle difference between these 
two operations. I am not sure if workflow dir only contains action dir though. 
But arguably, it might be fine/better to clean up each action's dir if needed 
as soon as the action is done. So I slightly prefer to leave it as is to be 
more cautious.

Please take a look and let me know if you have other concerns.

> oozie.action.keep.action.dir not getting noticed
> 
>
> Key: OOZIE-2796
> URL: https://issues.apache.org/jira/browse/OOZIE-2796
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 4.3.0
>Reporter: Andras Piros
>Assignee: Xiaobin Zheng
> Fix For: trunk
>
> Attachments: OOZIE-2796-1.patch, OOZIE-2796-2.patch, 
> OOZIE-2796-3.patch, OOZIE-2796-4.patch
>
>
> When trying to add following configuration entry:
> {code:xml}
> 
> oozie.action.keep.action.dir
> true
> 
> {code}
> to the application level configuration of {{workflow.xml}}, the workflow 
> directories are removed just after the workflow has finished.
> This should not be the case.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] Subscription: Oozie Patch Available

2017-02-24 Thread jira 
Issue Subscription
Filter: Oozie Patch Available (86 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-2619  Make  Hive action defaults to match hive defaults when running from 
command line
https://issues-test.apache.org/jira/browse/OOZIE-2619
OOZIE-2618  Include hive-common in hive-beeline as compile component for 
hive-server2 (hive2 action)
https://issues-test.apache.org/jira/browse/OOZIE-2618
OOZIE-2617  Read properties files in action configurations
https://issues-test.apache.org/jira/browse/OOZIE-2617
OOZIE-2616  Add Tez profile for Hive and Pig sharelibs
https://issues-test.apache.org/jira/browse/OOZIE-2616
OOZIE-2606  Set spark.yarn.jars to fix Spark 2.0 with Oozie
https://issues-test.apache.org/jira/browse/OOZIE-2606
OOZIE-2589  CompletedActionXCommand priority param is of no effect
https://issues-test.apache.org/jira/browse/OOZIE-2589
OOZIE-2588  Support getting credentials for same cluster hcat when credentials 
config is empty
https://issues-test.apache.org/jira/browse/OOZIE-2588
OOZIE-2585  Disable or remove test case 
TestPartitionDependencyManagerEhcache.testMemoryUsageAndSpeedOverflowToDisk and 
testMemoryUsageAndSpeed
https://issues-test.apache.org/jira/browse/OOZIE-2585
OOZIE-2584  Eliminate Thread.sleep() calls in TestMemoryLocks
https://issues-test.apache.org/jira/browse/OOZIE-2584
OOZIE-2583  oozie throws EL Exception when reference variable name containing 
dot
https://issues-test.apache.org/jira/browse/OOZIE-2583
OOZIE-2582  Populating external child Ids for action failures
https://issues-test.apache.org/jira/browse/OOZIE-2582
OOZIE-2573  dataset url  which contains spaces can not be handled rightly
https://issues-test.apache.org/jira/browse/OOZIE-2573
OOZIE-2572  SLA DURATION miss not shown when job is running for longer than 
expected time
https://issues-test.apache.org/jira/browse/OOZIE-2572
OOZIE-2571  Add scala.binary.version Maven property so that Scala 2.11 can be 
used
https://issues-test.apache.org/jira/browse/OOZIE-2571
OOZIE-2569  Adding yarn-site, core-site, hdfs-site and mapred-site into spark 
launcher 
https://issues-test.apache.org/jira/browse/OOZIE-2569
OOZIE-2568  SSH  action pretends to retry automaticly when it failed
https://issues-test.apache.org/jira/browse/OOZIE-2568
OOZIE-2564  Create new log4j config for unit tests so that logged messages 
contain threads
https://issues-test.apache.org/jira/browse/OOZIE-2564
OOZIE-2552  Update ActiveMQ version for security and other fixes
https://issues-test.apache.org/jira/browse/OOZIE-2552
OOZIE-2538  Update HttpClient versions to close security vulnerabilities
https://issues-test.apache.org/jira/browse/OOZIE-2538
OOZIE-2534  Versioned action libs (similar to how SharedLibs works)
https://issues-test.apache.org/jira/browse/OOZIE-2534
OOZIE-2530  Remove jline from parent and sharelib pig pom
https://issues-test.apache.org/jira/browse/OOZIE-2530
OOZIE-2528  Print out environment variables in LauncherMapper
https://issues-test.apache.org/jira/browse/OOZIE-2528
OOZIE-2521  Filter options are case sensitive for jobtye=bundle
https://issues-test.apache.org/jira/browse/OOZIE-2521
OOZIE-2518  Oozie is creating many files and directories in /tmp per day
https://issues-test.apache.org/jira/browse/OOZIE-2518
OOZIE-2517  Add support for startCreatedTime and endCreatedTime filters for 
coord and bundles
https://issues-test.apache.org/jira/browse/OOZIE-2517
OOZIE-2513  log.scan.duration should not be used for error and audit logs
https://issues-test.apache.org/jira/browse/OOZIE-2513
OOZIE-2512  ShareLibservice returns incorrect path for jar
https://issues-test.apache.org/jira/browse/OOZIE-2512
OOZIE-2508  Documentation change for Coord action rerun [OOZIE-1735]
https://issues-test.apache.org/jira/browse/OOZIE-2508
OOZIE-2501  ZK reentrant lock doesn't work for few cases
https://issues-test.apache.org/jira/browse/OOZIE-2501
OOZIE-2500  -DtestJarSimple option mentioned in minioozie doc does not work
https://issues-test.apache.org/jira/browse/OOZIE-2500
OOZIE-2499  map-reduce launcher does not need distributed files, archives 
except  jar of input/outputformat class
https://issues-test.apache.org/jira/browse/OOZIE-2499
OOZIE-2498  Oozie CallerId configuration for downstream components 
https://issues-test.apache.org/jira/browse/OOZIE-2498
OOZIE-2495  change action status from  ErrorType.NON_TRANSIENT to TRANSIENT 
when SSH action occurs AUTH_FAILED occasionally
https://issues-test.apache.org/jira/browse/OOZIE-2495
OOZIE-2491  oozie acl cannot specify group,it does`t work
https://issues-test.apache.org/jira/browse/OOZIE-2491

[jira] [Commented] (OOZIE-2792) Hive2 action is not parsing Spark application ID from log file properly when Hive is on Spark

2017-02-24 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883888#comment-15883888
 ] 

Hadoop QA commented on OOZIE-2792:
--

Testing JIRA OOZIE-2792

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in 
[hadooplibs/hadoop-utils-2].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1886
.Tests failed: 1
.Tests errors: 0

.The patch failed the following testcases:

.  testFsFailover(org.apache.oozie.action.TestActionFailover)

.Tests failing with errors:
.  

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/3660/

> Hive2 action is not parsing Spark application ID from log file properly when 
> Hive is on Spark
> -
>
> Key: OOZIE-2792
> URL: https://issues.apache.org/jira/browse/OOZIE-2792
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Reporter: Xiaobin Zheng
>Assignee: Xiaobin Zheng
>Priority: Minor
> Fix For: 5.0.0
>
> Attachments: OOZIE-2792-1.patch, OOZIE-2792-2.patch, 
> OOZIE-2792-3.patch, OOZIE-2792-4-amendment.patch, OOZIE-2792-4.patch
>
>
> When Hive2 is on Spark, hive action is not able to parse Spark application ID 
> from log file as 'externalChildID' like Spark/MR actions. This makes it hard 
> to tell which job hive launches from Oozie server for a particular workflow.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2792) Hive2 action is not parsing Spark application ID from log file properly when Hive is on Spark

2017-02-24 Thread Xiaobin Zheng (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobin Zheng updated OOZIE-2792:
-
Attachment: OOZIE-2792-4-amendment.patch

This amendment patch is made on top of OOZIE-2792-4.patch, updating the regex 
used to parse Spark application ID.

> Hive2 action is not parsing Spark application ID from log file properly when 
> Hive is on Spark
> -
>
> Key: OOZIE-2792
> URL: https://issues.apache.org/jira/browse/OOZIE-2792
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Reporter: Xiaobin Zheng
>Assignee: Xiaobin Zheng
>Priority: Minor
> Fix For: 5.0.0
>
> Attachments: OOZIE-2792-1.patch, OOZIE-2792-2.patch, 
> OOZIE-2792-3.patch, OOZIE-2792-4-amendment.patch, OOZIE-2792-4.patch
>
>
> When Hive2 is on Spark, hive action is not able to parse Spark application ID 
> from log file as 'externalChildID' like Spark/MR actions. This makes it hard 
> to tell which job hive launches from Oozie server for a particular workflow.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Reopened] (OOZIE-2792) Hive2 action is not parsing Spark application ID from log file properly when Hive is on Spark

2017-02-24 Thread Xiaobin Zheng (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobin Zheng reopened OOZIE-2792:
--

Reopen the issue. Previous regex used to parse Spark application ID is 
inaccurate. I will post an amendment patch.

> Hive2 action is not parsing Spark application ID from log file properly when 
> Hive is on Spark
> -
>
> Key: OOZIE-2792
> URL: https://issues.apache.org/jira/browse/OOZIE-2792
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Reporter: Xiaobin Zheng
>Assignee: Xiaobin Zheng
>Priority: Minor
> Fix For: 5.0.0
>
> Attachments: OOZIE-2792-1.patch, OOZIE-2792-2.patch, 
> OOZIE-2792-3.patch, OOZIE-2792-4.patch
>
>
> When Hive2 is on Spark, hive action is not able to parse Spark application ID 
> from log file as 'externalChildID' like Spark/MR actions. This makes it hard 
> to tell which job hive launches from Oozie server for a particular workflow.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain

2017-02-24 Thread Robert Kanter (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883443#comment-15883443
 ] 

Robert Kanter edited comment on OOZIE-2803 at 2/24/17 8:14 PM:
---

+1

I also tried this out in a cluster


was (Author: rkanter):
+1

> Mask passwords when printing out configs/args in MapReduceMain and SparkMain
> 
>
> Key: OOZIE-2803
> URL: https://issues.apache.org/jira/browse/OOZIE-2803
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch, 
> OOZIE-2803-003.patch, OOZIE-2803-004.patch, OOZIE-2803-005.patch
>
>
> Sometimes passwords are displayed in both MapReduce and Spark action.
> *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to 
> some Hadoop-specific config values, like {{mapred.child.env}}. This is easy 
> to fix because we already have a method {{logMasking()}} where you can define 
> a {{maskSet}} which contains a list of property keys to be masked.
> Note that this is not necessarily the perfect solution, since you can pass 
> multiple env. vars separated by a colon, and only the password specific parts 
> should be masked. But we need a working solution relatively quickly - later 
> we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right 
> now it only works with {{Map}}).
> *Spark*:  for Spark, we have to pass passwords like this:
> {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}}
> The Spark arguments are printed in {{SparkMain.run()}}. There is already a 
> code in {{LauncherMapper.printArgs()}} which deals with situations like this, 
> but it's not perfect because it only works if the args look something like 
> {{--password pwd123}}. So if a single arg contains a password, it doesn't 
> work, therefore we need a different approach here.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain

2017-02-24 Thread Robert Kanter (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883443#comment-15883443
 ] 

Robert Kanter commented on OOZIE-2803:
--

+1

> Mask passwords when printing out configs/args in MapReduceMain and SparkMain
> 
>
> Key: OOZIE-2803
> URL: https://issues.apache.org/jira/browse/OOZIE-2803
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch, 
> OOZIE-2803-003.patch, OOZIE-2803-004.patch, OOZIE-2803-005.patch
>
>
> Sometimes passwords are displayed in both MapReduce and Spark action.
> *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to 
> some Hadoop-specific config values, like {{mapred.child.env}}. This is easy 
> to fix because we already have a method {{logMasking()}} where you can define 
> a {{maskSet}} which contains a list of property keys to be masked.
> Note that this is not necessarily the perfect solution, since you can pass 
> multiple env. vars separated by a colon, and only the password specific parts 
> should be masked. But we need a working solution relatively quickly - later 
> we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right 
> now it only works with {{Map}}).
> *Spark*:  for Spark, we have to pass passwords like this:
> {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}}
> The Spark arguments are printed in {{SparkMain.run()}}. There is already a 
> code in {{LauncherMapper.printArgs()}} which deals with situations like this, 
> but it's not perfect because it only works if the args look something like 
> {{--password pwd123}}. So if a single arg contains a password, it doesn't 
> work, therefore we need a different approach here.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2457) Oozie log parsing regex consume more than 90% cpu

2017-02-24 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883438#comment-15883438
 ] 

Hadoop QA commented on OOZIE-2457:
--

Testing JIRA OOZIE-2457

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:red}-1{color} the patch contains 1 line(s) with trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in 
[hadooplibs/hadoop-utils-2].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1879
.Tests failed: 4
.Tests errors: 0

.The patch failed the following testcases:

.  
testProcessRemainingLog(org.apache.oozie.util.TestTimestampedMessageParser)
.  testFsFailover(org.apache.oozie.action.TestActionFailover)
.  testloglevel_Error(org.apache.oozie.util.TestXLogUserFilterParam)
.  testConnectionDrop(org.apache.oozie.jms.TestJMSJobEventListener)

.Tests failing with errors:
.  

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/3659/

> Oozie log parsing regex consume more than 90% cpu
> -
>
> Key: OOZIE-2457
> URL: https://issues.apache.org/jira/browse/OOZIE-2457
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-2457-1.patch, OOZIE-2457-2.patch, 
> OOZIE-2457-3.patch, OOZIE-2457-4.patch
>
>
> http-0.0.0.0-4080-26  TID=62215  STATE=RUNNABLE  CPU_TIME=1992 (92.59%)  
> USER_TIME=1990 (92.46%) Allocted: 269156584
> java.util.regex.Pattern$Curly.match0(Pattern.java:4170)
> java.util.regex.Pattern$Curly.match(Pattern.java:4132)
> java.util.regex.Pattern$GroupHead.match(Pattern.java:4556)
> java.util.regex.Matcher.match(Matcher.java:1221)
> java.util.regex.Matcher.matches(Matcher.java:559)
> org.apache.oozie.util.XLogFilter.matches(XLogFilter.java:136)
> 
> org.apache.oozie.util.TimestampedMessageParser.parseNextLine(TimestampedMessageParser.java:145)
> 
> org.apache.oozie.util.TimestampedMessageParser.increment(TimestampedMessageParser.java:92)
> Regex 
> {code}
> (.* USER\[[^\]]*\] GROUP\[[^\]]*\] TOKEN\[[^\]]*\] APP\[[^\]]*\] 
> JOB\[000-150625114739728-oozie-puru-W\] ACTION\[[^\]]*\] .*)
> {code}
> For single line parsing we use two regex.
> 1. 
> {code}
> public ArrayList splitLogMessage(String logLine) {
>

Build failed in Jenkins: oozie-trunk-precommit-build #3659

2017-02-24 Thread Apache Jenkins Server 
See 


--
[...truncated 1.54 MB...]
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/HadoopELFunctions.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/CredentialsProperties.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/MapReduceActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/CredentialsProvider.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/HbaseCredentials.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/FsActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/Hive2ActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/SparkActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/HiveActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/OozieJobInfo.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/ActionExecutor.Context.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/CoordinatorEngine.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/MessageReceiver.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/DefaultConnectionContext.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/ConnectionContext.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/JMSExceptionListener.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/JMSConnectionInfo.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/JMSJobEventListener.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/MessageHandler.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/JMSSLAEventListener.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/dependency/URIHandlerException.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/dependency/URIHandler.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/dependency/DependencyChecker.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/dependency/hcat/WaitingActions.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/dependency/hcat/SimpleHCatDependencyCache.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/dependency/hcat/EhcacheHCatDependencyCache.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/dependency/hcat/HCatDependencyCache.html
 longer than 100 characters.
[WARNING] Entry:

[jira] [Updated] (OOZIE-2457) Oozie log parsing regex consume more than 90% cpu

2017-02-24 Thread Satish Subhashrao Saley (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-2457:
---
Attachment: OOZIE-2457-4.patch

> Oozie log parsing regex consume more than 90% cpu
> -
>
> Key: OOZIE-2457
> URL: https://issues.apache.org/jira/browse/OOZIE-2457
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-2457-1.patch, OOZIE-2457-2.patch, 
> OOZIE-2457-3.patch, OOZIE-2457-4.patch
>
>
> http-0.0.0.0-4080-26  TID=62215  STATE=RUNNABLE  CPU_TIME=1992 (92.59%)  
> USER_TIME=1990 (92.46%) Allocted: 269156584
> java.util.regex.Pattern$Curly.match0(Pattern.java:4170)
> java.util.regex.Pattern$Curly.match(Pattern.java:4132)
> java.util.regex.Pattern$GroupHead.match(Pattern.java:4556)
> java.util.regex.Matcher.match(Matcher.java:1221)
> java.util.regex.Matcher.matches(Matcher.java:559)
> org.apache.oozie.util.XLogFilter.matches(XLogFilter.java:136)
> 
> org.apache.oozie.util.TimestampedMessageParser.parseNextLine(TimestampedMessageParser.java:145)
> 
> org.apache.oozie.util.TimestampedMessageParser.increment(TimestampedMessageParser.java:92)
> Regex 
> {code}
> (.* USER\[[^\]]*\] GROUP\[[^\]]*\] TOKEN\[[^\]]*\] APP\[[^\]]*\] 
> JOB\[000-150625114739728-oozie-puru-W\] ACTION\[[^\]]*\] .*)
> {code}
> For single line parsing we use two regex.
> 1. 
> {code}
> public ArrayList splitLogMessage(String logLine) {
> Matcher splitter = SPLITTER_PATTERN.matcher(logLine);
> if (splitter.matches()) {
> ArrayList logParts = new ArrayList();
> logParts.add(splitter.group(1));// timestamp
> logParts.add(splitter.group(2));// log level
> logParts.add(splitter.group(3));// Log Message
> return logParts;
> }
> else {
> return null;
> }
> }
> {code}
> 2.
> {code}
>  public boolean matches(ArrayList logParts) {
> if (getStartDate() != null) {
> if (logParts.get(0).substring(0, 
> 19).compareTo(getFormattedStartDate()) < 0) {
> return false;
> }
> }
> String logLevel = logParts.get(1);
> String logMessage = logParts.get(2);
> if (this.logLevels == null || 
> this.logLevels.containsKey(logLevel.toUpperCase())) {
> Matcher logMatcher = filterPattern.matcher(logMessage);
> return logMatcher.matches();
> }
> else {
> return false;
> }
> }
> {code}
> Also there is repetitive parsing  for same log message in
> {code}
> private String parseTimestamp(String line) {
> String timestamp = null;
> ArrayList logParts = filter.splitLogMessage(line);
> if (logParts != null) {
> timestamp = logParts.get(0);
> }
> return timestamp;
> }
> {code}
> where the {{line}} has already parsed using regex and we already know the 
> {{logParts}} if any.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 43970: Oozie log parsing regex consume more than 90% cpu

2017-02-24 Thread Satish Saley 


> On Sept. 13, 2016, 2:35 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/util/XLogFilter.java, line 213
> > 
> >
> > Shouldn't this be true?

I revisited this logic. I have renamed isSplit to isMatched, because it made 
more sense. We will set isMatched to true only if the log line matches the log 
filter pattern, otherwise we set it to false. If log line matches to the log 
filter pattern, then we cut the message into three parts and recored those 
parts in a list to avoid regex matching further in the code.


> On Sept. 13, 2016, 2:35 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/util/TimestampedMessageParser.java, 
> > line 94
> > 
> >
> > When you rebase the patch, this is where you'll see a trivial conflict.

I saw this. I will take care of it while committing.


- Satish


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43970/#review148786
---


On June 14, 2016, 4:15 p.m., Satish Saley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43970/
> ---
> 
> (Updated June 14, 2016, 4:15 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: https://issues.apache.org/jira/browse/OOZIE-2417
> 
> https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/OOZIE-2417
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> This patch tries to minimize the parsing of same log line for multiple times 
> using different regex.
> It also caches the log parts once we figure out what those are, thus avoiding 
> the re-parsing of log line to get log parts whenever needed.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/util/LogLine.java PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/util/SimpleTimestampedMessageParser.java 
> 78cb042 
>   core/src/main/java/org/apache/oozie/util/TimestampedMessageParser.java 
> f691409 
>   core/src/main/java/org/apache/oozie/util/XLogFilter.java 3b49f77 
> 
> Diff: https://reviews.apache.org/r/43970/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Satish Saley
> 
>

[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain

2017-02-24 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882821#comment-15882821
 ] 

Hadoop QA commented on OOZIE-2803:
--

Testing JIRA OOZIE-2803

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 11 line(s) longer than 132 
characters
.{color:green}+1{color} the patch does adds/modifies 3 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:orange}0{color} There are [1] new bugs found in total that would be nice 
to have fixed.
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:orange}0{color} There are [1] new bugs found in [sharelib/oozie] 
that would be nice to have fixed.
.You can find the FindBugs diff here: sharelib/oozie/findbugs-new.html
.{color:green}+1{color} There are no new bugs found in 
[hadooplibs/hadoop-utils-2].
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in [examples].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1886
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/3658/

> Mask passwords when printing out configs/args in MapReduceMain and SparkMain
> 
>
> Key: OOZIE-2803
> URL: https://issues.apache.org/jira/browse/OOZIE-2803
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch, 
> OOZIE-2803-003.patch, OOZIE-2803-004.patch, OOZIE-2803-005.patch
>
>
> Sometimes passwords are displayed in both MapReduce and Spark action.
> *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to 
> some Hadoop-specific config values, like {{mapred.child.env}}. This is easy 
> to fix because we already have a method {{logMasking()}} where you can define 
> a {{maskSet}} which contains a list of property keys to be masked.
> Note that this is not necessarily the perfect solution, since you can pass 
> multiple env. vars separated by a colon, and only the password specific parts 
> should be masked. But we need a working solution relatively quickly - later 
> we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right 
> now it only works with {{Map}}).
> *Spark*:  for Spark, we have to pass passwords like this:
> {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}}
> The Spark arguments are printed in {{SparkMain.run()}}. There is already a 
> code in {{LauncherMapper.printArgs()}} which deals with situations like this, 
> but it's not perfect because it only works if the args look something like 
> {{--password pwd123}}. So if a single arg contains a password, it

Build failed in Jenkins: oozie-trunk-precommit-build #3658

2017-02-24 Thread Apache Jenkins Server 
See 


--
[...truncated 1.53 MB...]
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/MessageHandler.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/store/Store.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/store/StoreStatusFilter.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/store/OozieSchema.OozieColumn.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/store/OozieSchema.OozieTable.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/store/SLAStore.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/store/OozieSchema.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/store/WorkflowStore.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/store/StoreException.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/store/OozieSchema.OozieIndex.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/sla/SLACalculator.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/sla/SLACalculatorMemory.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/sla/SLAXCommandFactory.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/sla/service/SLAService.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/sla/listener/SLAJobEventListener.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/sla/listener/SLAEventListener.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/sla/listener/SLAEmailEventListener.EmailField.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/sla/listener/SLAEmailEventListener.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/sla/SLACalcStatus.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/sla/SLASummaryBean.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/sla/SLAOperations.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/sla/SLARegistrationBean.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/sla/SLACalculatorMemory.HistoryPurgeWorker.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/CoordinatorJobInfo.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/CoordinatorActionInfo.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/CoordinatorEngine.FILTER_COMPARATORS.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/BinaryBlob.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/BundleJobInfo.html
 longer than 100 characters.
[WARNING] Entry:

[jira] [Created] (OOZIE-2809) Oozie HA with secure zookeeper access fails because ZKUtils#setJaasConfiguration does not replace "_HOST" in principal string

2017-02-24 Thread Thomas Larsson (JIRA) 
Thomas Larsson created OOZIE-2809:
-

 Summary: Oozie HA with secure zookeeper access fails because 
ZKUtils#setJaasConfiguration does not replace "_HOST" in principal string
 Key: OOZIE-2809
 URL: https://issues.apache.org/jira/browse/OOZIE-2809
 Project: Oozie
  Issue Type: Bug
  Components: core
Affects Versions: 4.3.0, 4.2.0, trunk
Reporter: Thomas Larsson


Setting "oozie.zookeeper.secure" = true causes the oozie server not to start if 
"oozie.service.HadoopAccessorService.kerberos.principal" is defined with a 
"_HOST" token instead of an actual hostname.

The symptom when this happens is that the oozie log shows something like this:
{noformat}
org.apache.oozie.service.ServiceException: E1700: Issue communicating with 
ZooKeeper: KeeperErrorCode = NoAuth for /oozie/services/servers
at org.apache.oozie.service.ZKLocksService.init(ZKLocksService.java:76)
at 
org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
at org.apache.oozie.service.Services.setService(Services.java:372)
at org.apache.oozie.service.Services.loadServices(Services.java:305)
at org.apache.oozie.service.Services.init(Services.java:213)
at 
org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:46)
at 
org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4210)
at 
org.apache.catalina.core.StandardContext.start(StandardContext.java:4709)
at 
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:802)
at 
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583)
at 
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:676)
at 
org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:602)
at 
org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:503)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322)
at 
org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325)
at 
org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1068)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:822)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1060)
at 
org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
at 
org.apache.catalina.core.StandardService.start(StandardService.java:525)
at 
org.apache.catalina.core.StandardServer.start(StandardServer.java:759)
at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: org.apache.zookeeper.KeeperException$NoAuthException: 
KeeperErrorCode = NoAuth for /oozie/services/servers
at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:199)
at 
org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:682)
at 
org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:660)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
at 
org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:656)
at 
org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:441)
at 
org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:431)
at 
org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
at 
org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.internalRegisterService(ServiceDiscoveryImpl.java:176)
at 
org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.registerService(ServiceDiscoveryImpl.java:150)
at org.apache.oozie.util.ZKUtils.advertiseService(ZKUtils.java:217)
at org.apache.oozie.util.ZKUtils.(ZKUtils.java:141)
at org.apache.oozie.util.ZKUtils.register(ZKUtils.java:154)
at

[jira] [Updated] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain

2017-02-24 Thread Peter Bacsko (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Bacsko updated OOZIE-2803:

Attachment: OOZIE-2803-005.patch

> Mask passwords when printing out configs/args in MapReduceMain and SparkMain
> 
>
> Key: OOZIE-2803
> URL: https://issues.apache.org/jira/browse/OOZIE-2803
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch, 
> OOZIE-2803-003.patch, OOZIE-2803-004.patch, OOZIE-2803-005.patch
>
>
> Sometimes passwords are displayed in both MapReduce and Spark action.
> *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to 
> some Hadoop-specific config values, like {{mapred.child.env}}. This is easy 
> to fix because we already have a method {{logMasking()}} where you can define 
> a {{maskSet}} which contains a list of property keys to be masked.
> Note that this is not necessarily the perfect solution, since you can pass 
> multiple env. vars separated by a colon, and only the password specific parts 
> should be masked. But we need a working solution relatively quickly - later 
> we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right 
> now it only works with {{Map}}).
> *Spark*:  for Spark, we have to pass passwords like this:
> {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}}
> The Spark arguments are printed in {{SparkMain.run()}}. There is already a 
> code in {{LauncherMapper.printArgs()}} which deals with situations like this, 
> but it's not perfect because it only works if the args look something like 
> {{--password pwd123}}. So if a single arg contains a password, it doesn't 
> work, therefore we need a different approach here.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain

2017-02-24 Thread Peter Bacsko (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882579#comment-15882579
 ] 

Peter Bacsko commented on OOZIE-2803:
-

There are no findbugs warnings, Raw patch analysis complains about lines that 
are too long. But those are just the csv files that contain test data.

I changed {{MapReduceMain}} and {{PasswordMasker}} as you requested. 


> Mask passwords when printing out configs/args in MapReduceMain and SparkMain
> 
>
> Key: OOZIE-2803
> URL: https://issues.apache.org/jira/browse/OOZIE-2803
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch, 
> OOZIE-2803-003.patch, OOZIE-2803-004.patch, OOZIE-2803-005.patch
>
>
> Sometimes passwords are displayed in both MapReduce and Spark action.
> *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to 
> some Hadoop-specific config values, like {{mapred.child.env}}. This is easy 
> to fix because we already have a method {{logMasking()}} where you can define 
> a {{maskSet}} which contains a list of property keys to be masked.
> Note that this is not necessarily the perfect solution, since you can pass 
> multiple env. vars separated by a colon, and only the password specific parts 
> should be masked. But we need a working solution relatively quickly - later 
> we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right 
> now it only works with {{Map}}).
> *Spark*:  for Spark, we have to pass passwords like this:
> {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}}
> The Spark arguments are printed in {{SparkMain.run()}}. There is already a 
> code in {{LauncherMapper.printArgs()}} which deals with situations like this, 
> but it's not perfect because it only works if the args look something like 
> {{--password pwd123}}. So if a single arg contains a password, it doesn't 
> work, therefore we need a different approach here.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)