[jira] Subscription: Oozie Patch Available

2017-02-21 Thread jira 
Issue Subscription
Filter: Oozie Patch Available (91 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-2803  Mask passwords when printing out configs/args in MapReduceMain and 
SparkMain
https://issues.apache.org/jira/browse/OOZIE-2803
OOZIE-2796  oozie.action.keep.action.dir not getting noticed
https://issues.apache.org/jira/browse/OOZIE-2796
OOZIE-2795  Create lib directory or symlink for Oozie CLI during packaging
https://issues.apache.org/jira/browse/OOZIE-2795
OOZIE-2791  ShareLib installation may fail on busy Hadoop clusters
https://issues.apache.org/jira/browse/OOZIE-2791
OOZIE-2785  Master is not compiling with hadoop-1 profile
https://issues.apache.org/jira/browse/OOZIE-2785
OOZIE-2784  Include WEEK as a parameter in the Coordinator Expression Language 
Evaulator
https://issues.apache.org/jira/browse/OOZIE-2784
OOZIE-2780  Upgrade hadoop-common version to 2.6.0
https://issues.apache.org/jira/browse/OOZIE-2780
OOZIE-2779  Mask Hive2 action Beeline JDBC password
https://issues.apache.org/jira/browse/OOZIE-2779
OOZIE-2769  Extend FS action to allow setrep on a file
https://issues.apache.org/jira/browse/OOZIE-2769
OOZIE-2753  Update Docs for Jetty Tomcat changes
https://issues.apache.org/jira/browse/OOZIE-2753
OOZIE-2751  LocalOozieClient is missing methods from OozieClient
https://issues.apache.org/jira/browse/OOZIE-2751
OOZIE-2750  Spelling errors in the log messages and exception messages
https://issues.apache.org/jira/browse/OOZIE-2750
OOZIE-2746  Several tests failure in TestV2ValidateServlet.java
https://issues.apache.org/jira/browse/OOZIE-2746
OOZIE-2739  Remove property expansion pattern from ShellMain's log4j properties 
content
https://issues.apache.org/jira/browse/OOZIE-2739
OOZIE-2736  Reduce the number of threads during test execution
https://issues.apache.org/jira/browse/OOZIE-2736
OOZIE-2733  change org.apache.hadoop.fs.permission.AccessControlException to 
org.apache.hadoop.security.AccessControlException
https://issues.apache.org/jira/browse/OOZIE-2733
OOZIE-2726  Flaky test due to daylight saving changes
https://issues.apache.org/jira/browse/OOZIE-2726
OOZIE-2718  Improve -dryrun for bundles
https://issues.apache.org/jira/browse/OOZIE-2718
OOZIE-2713  Provide an option to log out contents of the localized jar files
https://issues.apache.org/jira/browse/OOZIE-2713
OOZIE-2701  Oozie to support Multiple HCatalog URIs
https://issues.apache.org/jira/browse/OOZIE-2701
OOZIE-2694  Add logging for FsActionExecutor 
https://issues.apache.org/jira/browse/OOZIE-2694
OOZIE-2693  SimpleHCatDependencyCache.removeMissingDependency can throw NPE
https://issues.apache.org/jira/browse/OOZIE-2693
OOZIE-2692  Oozie job submit doesn't report error message to user if there is 
any issue with job conf
https://issues.apache.org/jira/browse/OOZIE-2692
OOZIE-2684  Bad database schema error for WF_ACTIONS table
https://issues.apache.org/jira/browse/OOZIE-2684
OOZIE-2681  fix javadoc to compile on JDK8 again
https://issues.apache.org/jira/browse/OOZIE-2681
OOZIE-2670  Upgrade Hbase to 1.2
https://issues.apache.org/jira/browse/OOZIE-2670
OOZIE-2668  Status update and recovery problems when coord action and its 
children not in sync
https://issues.apache.org/jira/browse/OOZIE-2668
OOZIE-2662  DB migration fails if DB is too big
https://issues.apache.org/jira/browse/OOZIE-2662
OOZIE-2652  Skip trash while deleting hive table partition
https://issues.apache.org/jira/browse/OOZIE-2652
OOZIE-2650  Retry coord start on database exceptions
https://issues.apache.org/jira/browse/OOZIE-2650
OOZIE-2644  Skip queuing Notification Commands when there's nothing to notify
https://issues.apache.org/jira/browse/OOZIE-2644
OOZIE-2641  rerunning an oozie coordinator job stucks on waiting
https://issues.apache.org/jira/browse/OOZIE-2641
OOZIE-2629  Get OozieShareLibCLI to perform a final rename to destPath when 
creating a Sharelib
https://issues.apache.org/jira/browse/OOZIE-2629
OOZIE-2618  Include hive-common in hive-beeline as compile component for 
hive-server2 (hive2 action)
https://issues.apache.org/jira/browse/OOZIE-2618
OOZIE-2608  Comma in oozie.service.JPAService.jdbc.password value results in 
authentication error
https://issues.apache.org/jira/browse/OOZIE-2608
OOZIE-2585  Disable or remove test case 
TestPartitionDependencyManagerEhcache.testMemoryUsageAndSpeedOverflowToDisk and 
testMemoryUsageAndSpeed
https://issues.apache.org/jira/browse/OOZIE-2585
OOZIE-2573  dataset url  which contains spaces can not be handled rightly
https://issues.apache.org/jira/browse/OOZIE-2573

[jira] Subscription: Oozie Patch Available

2017-02-21 Thread jira 
Issue Subscription
Filter: Oozie Patch Available (86 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-2619  Make  Hive action defaults to match hive defaults when running from 
command line
https://issues-test.apache.org/jira/browse/OOZIE-2619
OOZIE-2618  Include hive-common in hive-beeline as compile component for 
hive-server2 (hive2 action)
https://issues-test.apache.org/jira/browse/OOZIE-2618
OOZIE-2617  Read properties files in action configurations
https://issues-test.apache.org/jira/browse/OOZIE-2617
OOZIE-2616  Add Tez profile for Hive and Pig sharelibs
https://issues-test.apache.org/jira/browse/OOZIE-2616
OOZIE-2606  Set spark.yarn.jars to fix Spark 2.0 with Oozie
https://issues-test.apache.org/jira/browse/OOZIE-2606
OOZIE-2589  CompletedActionXCommand priority param is of no effect
https://issues-test.apache.org/jira/browse/OOZIE-2589
OOZIE-2588  Support getting credentials for same cluster hcat when credentials 
config is empty
https://issues-test.apache.org/jira/browse/OOZIE-2588
OOZIE-2585  Disable or remove test case 
TestPartitionDependencyManagerEhcache.testMemoryUsageAndSpeedOverflowToDisk and 
testMemoryUsageAndSpeed
https://issues-test.apache.org/jira/browse/OOZIE-2585
OOZIE-2584  Eliminate Thread.sleep() calls in TestMemoryLocks
https://issues-test.apache.org/jira/browse/OOZIE-2584
OOZIE-2583  oozie throws EL Exception when reference variable name containing 
dot
https://issues-test.apache.org/jira/browse/OOZIE-2583
OOZIE-2582  Populating external child Ids for action failures
https://issues-test.apache.org/jira/browse/OOZIE-2582
OOZIE-2573  dataset url  which contains spaces can not be handled rightly
https://issues-test.apache.org/jira/browse/OOZIE-2573
OOZIE-2572  SLA DURATION miss not shown when job is running for longer than 
expected time
https://issues-test.apache.org/jira/browse/OOZIE-2572
OOZIE-2571  Add scala.binary.version Maven property so that Scala 2.11 can be 
used
https://issues-test.apache.org/jira/browse/OOZIE-2571
OOZIE-2569  Adding yarn-site, core-site, hdfs-site and mapred-site into spark 
launcher 
https://issues-test.apache.org/jira/browse/OOZIE-2569
OOZIE-2568  SSH  action pretends to retry automaticly when it failed
https://issues-test.apache.org/jira/browse/OOZIE-2568
OOZIE-2564  Create new log4j config for unit tests so that logged messages 
contain threads
https://issues-test.apache.org/jira/browse/OOZIE-2564
OOZIE-2552  Update ActiveMQ version for security and other fixes
https://issues-test.apache.org/jira/browse/OOZIE-2552
OOZIE-2538  Update HttpClient versions to close security vulnerabilities
https://issues-test.apache.org/jira/browse/OOZIE-2538
OOZIE-2534  Versioned action libs (similar to how SharedLibs works)
https://issues-test.apache.org/jira/browse/OOZIE-2534
OOZIE-2530  Remove jline from parent and sharelib pig pom
https://issues-test.apache.org/jira/browse/OOZIE-2530
OOZIE-2528  Print out environment variables in LauncherMapper
https://issues-test.apache.org/jira/browse/OOZIE-2528
OOZIE-2521  Filter options are case sensitive for jobtye=bundle
https://issues-test.apache.org/jira/browse/OOZIE-2521
OOZIE-2518  Oozie is creating many files and directories in /tmp per day
https://issues-test.apache.org/jira/browse/OOZIE-2518
OOZIE-2517  Add support for startCreatedTime and endCreatedTime filters for 
coord and bundles
https://issues-test.apache.org/jira/browse/OOZIE-2517
OOZIE-2513  log.scan.duration should not be used for error and audit logs
https://issues-test.apache.org/jira/browse/OOZIE-2513
OOZIE-2512  ShareLibservice returns incorrect path for jar
https://issues-test.apache.org/jira/browse/OOZIE-2512
OOZIE-2508  Documentation change for Coord action rerun [OOZIE-1735]
https://issues-test.apache.org/jira/browse/OOZIE-2508
OOZIE-2501  ZK reentrant lock doesn't work for few cases
https://issues-test.apache.org/jira/browse/OOZIE-2501
OOZIE-2500  -DtestJarSimple option mentioned in minioozie doc does not work
https://issues-test.apache.org/jira/browse/OOZIE-2500
OOZIE-2499  map-reduce launcher does not need distributed files, archives 
except  jar of input/outputformat class
https://issues-test.apache.org/jira/browse/OOZIE-2499
OOZIE-2498  Oozie CallerId configuration for downstream components 
https://issues-test.apache.org/jira/browse/OOZIE-2498
OOZIE-2495  change action status from  ErrorType.NON_TRANSIENT to TRANSIENT 
when SSH action occurs AUTH_FAILED occasionally
https://issues-test.apache.org/jira/browse/OOZIE-2495
OOZIE-2491  oozie acl cannot specify group,it does`t work
https://issues-test.apache.org/jira/browse/OOZIE-2491

[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain

2017-02-21 Thread Abhishek Bafna (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877556#comment-15877556
 ] 

Abhishek Bafna commented on OOZIE-2803:
---

Few points: 
There are some unused imports in the {{MapReduceMain}}.
Fix the java doc for {{PasswordMasker:maskPasswordsIfNecessary}} and 
{{PasswordMasker:mask}}.
All the field members of {{PasswordMasker}} are declared final and class in 
itself does not have any state. it can be converted into Utility Class.

> Mask passwords when printing out configs/args in MapReduceMain and SparkMain
> 
>
> Key: OOZIE-2803
> URL: https://issues.apache.org/jira/browse/OOZIE-2803
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch, 
> OOZIE-2803-003.patch
>
>
> Sometimes passwords are displayed in both MapReduce and Spark action.
> *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to 
> some Hadoop-specific config values, like {{mapred.child.env}}. This is easy 
> to fix because we already have a method {{logMasking()}} where you can define 
> a {{maskSet}} which contains a list of property keys to be masked.
> Note that this is not necessarily the perfect solution, since you can pass 
> multiple env. vars separated by a colon, and only the password specific parts 
> should be masked. But we need a working solution relatively quickly - later 
> we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right 
> now it only works with {{Map}}).
> *Spark*:  for Spark, we have to pass passwords like this:
> {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}}
> The Spark arguments are printed in {{SparkMain.run()}}. There is already a 
> code in {{LauncherMapper.printArgs()}} which deals with situations like this, 
> but it's not perfect because it only works if the args look something like 
> {{--password pwd123}}. So if a single arg contains a password, it doesn't 
> work, therefore we need a different approach here.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain

2017-02-21 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876469#comment-15876469
 ] 

Hadoop QA commented on OOZIE-2803:
--

Testing JIRA OOZIE-2803

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 11 line(s) longer than 132 
characters
.{color:green}+1{color} the patch does adds/modifies 3 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:orange}0{color} There are [1] new bugs found in total that would be nice 
to have fixed.
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in 
[hadooplibs/hadoop-utils-2].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:orange}0{color} There are [1] new bugs found in [sharelib/oozie] 
that would be nice to have fixed.
.You can find the FindBugs diff here: sharelib/oozie/findbugs-new.html
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1886
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/3653/

> Mask passwords when printing out configs/args in MapReduceMain and SparkMain
> 
>
> Key: OOZIE-2803
> URL: https://issues.apache.org/jira/browse/OOZIE-2803
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch, 
> OOZIE-2803-003.patch
>
>
> Sometimes passwords are displayed in both MapReduce and Spark action.
> *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to 
> some Hadoop-specific config values, like {{mapred.child.env}}. This is easy 
> to fix because we already have a method {{logMasking()}} where you can define 
> a {{maskSet}} which contains a list of property keys to be masked.
> Note that this is not necessarily the perfect solution, since you can pass 
> multiple env. vars separated by a colon, and only the password specific parts 
> should be masked. But we need a working solution relatively quickly - later 
> we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right 
> now it only works with {{Map}}).
> *Spark*:  for Spark, we have to pass passwords like this:
> {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}}
> The Spark arguments are printed in {{SparkMain.run()}}. There is already a 
> code in {{LauncherMapper.printArgs()}} which deals with situations like this, 
> but it's not perfect because it only works if the args look something like 
> {{--password pwd123}}. So if a single arg contains a password, it doesn't 
> work, therefore we need a different

[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain

2017-02-21 Thread Peter Bacsko (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876259#comment-15876259
 ] 

Peter Bacsko commented on OOZIE-2803:
-

Fixing Findbugs problems. The long lines (> 132 chars) occur only in the csv 
files, so this warning can be ignored.

> Mask passwords when printing out configs/args in MapReduceMain and SparkMain
> 
>
> Key: OOZIE-2803
> URL: https://issues.apache.org/jira/browse/OOZIE-2803
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch, 
> OOZIE-2803-003.patch
>
>
> Sometimes passwords are displayed in both MapReduce and Spark action.
> *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to 
> some Hadoop-specific config values, like {{mapred.child.env}}. This is easy 
> to fix because we already have a method {{logMasking()}} where you can define 
> a {{maskSet}} which contains a list of property keys to be masked.
> Note that this is not necessarily the perfect solution, since you can pass 
> multiple env. vars separated by a colon, and only the password specific parts 
> should be masked. But we need a working solution relatively quickly - later 
> we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right 
> now it only works with {{Map}}).
> *Spark*:  for Spark, we have to pass passwords like this:
> {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}}
> The Spark arguments are printed in {{SparkMain.run()}}. There is already a 
> code in {{LauncherMapper.printArgs()}} which deals with situations like this, 
> but it's not perfect because it only works if the args look something like 
> {{--password pwd123}}. So if a single arg contains a password, it doesn't 
> work, therefore we need a different approach here.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain

2017-02-21 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876043#comment-15876043
 ] 

Hadoop QA commented on OOZIE-2803:
--

Testing JIRA OOZIE-2803

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 12 line(s) longer than 132 
characters
.{color:green}+1{color} the patch does adds/modifies 3 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:red}-1{color} There are [2] new bugs found below threshold in total that 
must be fixed.
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in 
[hadooplibs/hadoop-utils-2].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:red}-1{color} There are [2] new bugs found below threshold in 
[sharelib/oozie] that must be fixed.
.You can find the FindBugs diff here (look for the red and orange ones): 
sharelib/oozie/findbugs-new.html
.The most important FindBugs errors are:
.At PasswordMasker.java:[lines 85-92]: key must be nonnull but is marked as 
nullable
.At PasswordMasker.java:[lines 85-92]: value must be nonnull but is marked 
as nullable
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1886
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/3652/

> Mask passwords when printing out configs/args in MapReduceMain and SparkMain
> 
>
> Key: OOZIE-2803
> URL: https://issues.apache.org/jira/browse/OOZIE-2803
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch
>
>
> Sometimes passwords are displayed in both MapReduce and Spark action.
> *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to 
> some Hadoop-specific config values, like {{mapred.child.env}}. This is easy 
> to fix because we already have a method {{logMasking()}} where you can define 
> a {{maskSet}} which contains a list of property keys to be masked.
> Note that this is not necessarily the perfect solution, since you can pass 
> multiple env. vars separated by a colon, and only the password specific parts 
> should be masked. But we need a working solution relatively quickly - later 
> we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right 
> now it only works with {{Map}}).
> *Spark*:  for Spark, we have to pass passwords like this:
> {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}}
> The Spark arguments are printed in {{SparkMain.run()}}. There is already a 
> code in

Build failed in Jenkins: oozie-trunk-precommit-build #3652

2017-02-21 Thread Apache Jenkins Server 
See 


--
[...truncated 1.53 MB...]
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/ShellActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/PigActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/HCatCredentials.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/JavaActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/CredentialException.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/FsELFunctions.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/DistcpActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/HadoopELFunctions.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/CredentialsProperties.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/MapReduceActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/CredentialsProvider.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/HbaseCredentials.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/FsActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/Hive2ActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/SparkActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/HiveActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/OozieJobInfo.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/ActionExecutor.Context.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/CoordinatorEngine.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/MessageReceiver.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/DefaultConnectionContext.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/ConnectionContext.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/JMSExceptionListener.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/JMSConnectionInfo.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/JMSJobEventListener.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/MessageHandler.html
 longer than 100 characters.
[WARNING] Entry:

[jira] [Commented] (OOZIE-2779) Mask Hive2 action Beeline JDBC password

2017-02-21 Thread Abhishek Bafna (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875887#comment-15875887
 ] 

Abhishek Bafna commented on OOZIE-2779:
---

[~rkanter] can you please take a look?

> Mask Hive2 action Beeline JDBC password
> ---
>
> Key: OOZIE-2779
> URL: https://issues.apache.org/jira/browse/OOZIE-2779
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
>  Labels: security
> Fix For: 5.0.0
>
> Attachments: OOZIE-2779-00.patch
>
>
> Hive2 Oozie launcher job prints the JDBC password into launcher stdout logs.
> {noformat}
> Beeline command arguments :
>  -u
>  jdbc:hive2://source-1:1/default
>  -n
>  ambari-qa
>  -p
>  DUMMY
>  -d
>  org.apache.hive.jdbc.HiveDriver
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2684) Bad database schema error for WF_ACTIONS table

2017-02-21 Thread Abhishek Bafna (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875885#comment-15875885
 ] 

Abhishek Bafna commented on OOZIE-2684:
---

[~rkanter] Can you please take a look? Thanks.

> Bad database schema error for WF_ACTIONS table
> --
>
> Key: OOZIE-2684
> URL: https://issues.apache.org/jira/browse/OOZIE-2684
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-2684-00.patch
>
>
> In {{SchemaCheckerService}}, Oozie compares expected and found indexed 
> columns and generates the below error message which could be confusing to the 
> users.
> {noformat}
> 2016-09-16 12:39:26,564  INFO SchemaCheckXCommand:520 - 
> SERVER[c6402.ambari.apache.org] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] 
> ACTION[-] About to check database schema
> 2016-09-16 12:39:26,703 ERROR SchemaCheckXCommand:517 - 
> SERVER[c6402.ambari.apache.org] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] 
> ACTION[-] Found [1] extra indexes for columns in table [WF_ACTIONS]: [wf_id]
> 2016-09-16 12:39:26,723 ERROR SchemaCheckXCommand:517 - 
> SERVER[c6402.ambari.apache.org] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] 
> ACTION[-] Database schema is BAD! Check previous error log messages for 
> details
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : oozie-trunk-find-patches-available #150495

2017-02-21 Thread Apache Jenkins Server 
See

Build failed in Jenkins: oozie-trunk-find-patches-available #150494

2017-02-21 Thread Apache Jenkins Server 
See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on H1 (Hadoop) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/oozie.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/oozie.git
 > git --version # timeout=10
 > git fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/oozie.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 77817e73588f665cbe43b0971cd93465d40711fb 
(refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 77817e73588f665cbe43b0971cd93465d40711fb
 > git rev-list 77817e73588f665cbe43b0971cd93465d40711fb # timeout=10
[oozie-trunk-find-patches-available] $ /bin/bash 
/tmp/hudson7050241608620948755.sh
mkdir: cannot create directory 
':
 File exists
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0100 
 335k0  335k0 0   334k  0 --:--:--  0:00:01 --:--:--  334k100  
919k0  919k0 0   413k  0 --:--:--  0:00:02 --:--:--  413k100  
919k0  919k0 0   413k  0 --:--:--  0:00:02 --:--:--  413k
curl: (18) transfer closed with outstanding read data remaining
Could not retrieve available patches from JIRA
Build step 'Execute shell' marked build as failure

[jira] [Updated] (OOZIE-2805) create derbyoozie

2017-02-21 Thread Abhishek Bafna (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Bafna updated OOZIE-2805:
--
Description: 
I guessed there is a bug because it generated "Exception" instead "Error" 
message.
I tried to use build Oozie package.
my Hadoop is installed on the remote server.
when I run
{code}
$oozie-4.3.0/distro/target/oozie-4.3.0-distro/oozie-4.3.0/bin/ooziedb.sh create 
-run
command I encountered below message:
  setting 
JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.131-2.6.9.0.el7_3.x86_64
  setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"

Validate DB Connection
Exception in thread "main" java.lang.NoClassDefFoundError: 
org/apache/hadoop/util/ReflectionUtils
at 
org.apache.oozie.service.Services.setServiceInternal(Services.java:377)
at org.apache.oozie.service.Services.(Services.java:111)
at org.apache.oozie.tools.OozieDBCLI.getJdbcConf(OozieDBCLI.java:165)
at 
org.apache.oozie.tools.OozieDBCLI.createConnection(OozieDBCLI.java:864)
at 
org.apache.oozie.tools.OozieDBCLI.validateConnection(OozieDBCLI.java:872)
at org.apache.oozie.tools.OozieDBCLI.createDB(OozieDBCLI.java:183)
at org.apache.oozie.tools.OozieDBCLI.run(OozieDBCLI.java:127)
at org.apache.oozie.tools.OozieDBCLI.main(OozieDBCLI.java:78)
Caused by: java.lang.ClassNotFoundException: 
org.apache.hadoop.util.ReflectionUtils
at java.net.URLClassLoader$1.run(URLClassLoader.java:359)
at java.net.URLClassLoader$1.run(URLClassLoader.java:348)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:347)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 8 more
{code}

  was:
I guessed there is a bug because it generated "Exception" instead "Error" 
message.
I tried to use build Oozie package.
my Hadoop is installed on the remote server.
when I run
oozie-4.3.0/distro/target/oozie-4.3.0-distro/oozie-4.3.0/bin/ooziedb.sh create 
-run
command I encountered below message:
  setting 
JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.131-2.6.9.0.el7_3.x86_64
  setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"

Validate DB Connection
Exception in thread "main" java.lang.NoClassDefFoundError: 
org/apache/hadoop/util/ReflectionUtils
at 
org.apache.oozie.service.Services.setServiceInternal(Services.java:377)
at org.apache.oozie.service.Services.(Services.java:111)
at org.apache.oozie.tools.OozieDBCLI.getJdbcConf(OozieDBCLI.java:165)
at 
org.apache.oozie.tools.OozieDBCLI.createConnection(OozieDBCLI.java:864)
at 
org.apache.oozie.tools.OozieDBCLI.validateConnection(OozieDBCLI.java:872)
at org.apache.oozie.tools.OozieDBCLI.createDB(OozieDBCLI.java:183)
at org.apache.oozie.tools.OozieDBCLI.run(OozieDBCLI.java:127)
at org.apache.oozie.tools.OozieDBCLI.main(OozieDBCLI.java:78)
Caused by: java.lang.ClassNotFoundException: 
org.apache.hadoop.util.ReflectionUtils
at java.net.URLClassLoader$1.run(URLClassLoader.java:359)
at java.net.URLClassLoader$1.run(URLClassLoader.java:348)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:347)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 8 more



> create derbyoozie
> -
>
> Key: OOZIE-2805
> URL: https://issues.apache.org/jira/browse/OOZIE-2805
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 4.3.0
> Environment: OS: CentOS 7
> used database is derbydb in oozie-default.xml.reference file
> user hduser and oozie added in Hadoop Hdfs config files.
>Reporter: Hossein Vatani
> Fix For: 4.3.0
>
>
> I guessed there is a bug because it generated "Exception" instead "Error" 
> message.
> I tried to use build Oozie package.
> my Hadoop is installed on the remote server.
> when I run
> {code}
> $oozie-4.3.0/distro/target/oozie-4.3.0-distro/oozie-4.3.0/bin/ooziedb.sh 
> create -run
> command I encountered below message:
>   setting 
> JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.131-2.6.9.0.el7_3.x86_64
>   setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
> Validate DB Connection
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/hadoop/util/ReflectionUtils
>   at 
> org.apache.oozie.service.Services.setServiceInternal(Services.java:377)
>   at org.apache.oozie.service.Services.(Services.java:111)
>   at

[jira] [Created] (OOZIE-2805) create derbyoozie

2017-02-21 Thread Hossein Vatani (JIRA) 
Hossein Vatani created OOZIE-2805:
-

 Summary: create derbyoozie
 Key: OOZIE-2805
 URL: https://issues.apache.org/jira/browse/OOZIE-2805
 Project: Oozie
  Issue Type: Bug
Affects Versions: 4.3.0
 Environment: OS: CentOS 7
used database is derbydb in oozie-default.xml.reference file
user hduser and oozie added in Hadoop Hdfs config files.
Reporter: Hossein Vatani
 Fix For: 4.3.0


I tried to use build Oozie package.
my Hadoop is installed on the remote server.
when I run
oozie-4.3.0/distro/target/oozie-4.3.0-distro/oozie-4.3.0/bin/ooziedb.sh create 
-run
command I encountered below message:
  setting 
JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.131-2.6.9.0.el7_3.x86_64
  setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"

Validate DB Connection
Exception in thread "main" java.lang.NoClassDefFoundError: 
org/apache/hadoop/util/ReflectionUtils
at 
org.apache.oozie.service.Services.setServiceInternal(Services.java:377)
at org.apache.oozie.service.Services.(Services.java:111)
at org.apache.oozie.tools.OozieDBCLI.getJdbcConf(OozieDBCLI.java:165)
at 
org.apache.oozie.tools.OozieDBCLI.createConnection(OozieDBCLI.java:864)
at 
org.apache.oozie.tools.OozieDBCLI.validateConnection(OozieDBCLI.java:872)
at org.apache.oozie.tools.OozieDBCLI.createDB(OozieDBCLI.java:183)
at org.apache.oozie.tools.OozieDBCLI.run(OozieDBCLI.java:127)
at org.apache.oozie.tools.OozieDBCLI.main(OozieDBCLI.java:78)
Caused by: java.lang.ClassNotFoundException: 
org.apache.hadoop.util.ReflectionUtils
at java.net.URLClassLoader$1.run(URLClassLoader.java:359)
at java.net.URLClassLoader$1.run(URLClassLoader.java:348)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:347)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 8 more




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)