[jira] Subscription: Oozie Patch Available

2017-02-23 Thread jira 
Issue Subscription
Filter: Oozie Patch Available (86 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-2619  Make  Hive action defaults to match hive defaults when running from 
command line
https://issues-test.apache.org/jira/browse/OOZIE-2619
OOZIE-2618  Include hive-common in hive-beeline as compile component for 
hive-server2 (hive2 action)
https://issues-test.apache.org/jira/browse/OOZIE-2618
OOZIE-2617  Read properties files in action configurations
https://issues-test.apache.org/jira/browse/OOZIE-2617
OOZIE-2616  Add Tez profile for Hive and Pig sharelibs
https://issues-test.apache.org/jira/browse/OOZIE-2616
OOZIE-2606  Set spark.yarn.jars to fix Spark 2.0 with Oozie
https://issues-test.apache.org/jira/browse/OOZIE-2606
OOZIE-2589  CompletedActionXCommand priority param is of no effect
https://issues-test.apache.org/jira/browse/OOZIE-2589
OOZIE-2588  Support getting credentials for same cluster hcat when credentials 
config is empty
https://issues-test.apache.org/jira/browse/OOZIE-2588
OOZIE-2585  Disable or remove test case 
TestPartitionDependencyManagerEhcache.testMemoryUsageAndSpeedOverflowToDisk and 
testMemoryUsageAndSpeed
https://issues-test.apache.org/jira/browse/OOZIE-2585
OOZIE-2584  Eliminate Thread.sleep() calls in TestMemoryLocks
https://issues-test.apache.org/jira/browse/OOZIE-2584
OOZIE-2583  oozie throws EL Exception when reference variable name containing 
dot
https://issues-test.apache.org/jira/browse/OOZIE-2583
OOZIE-2582  Populating external child Ids for action failures
https://issues-test.apache.org/jira/browse/OOZIE-2582
OOZIE-2573  dataset url  which contains spaces can not be handled rightly
https://issues-test.apache.org/jira/browse/OOZIE-2573
OOZIE-2572  SLA DURATION miss not shown when job is running for longer than 
expected time
https://issues-test.apache.org/jira/browse/OOZIE-2572
OOZIE-2571  Add scala.binary.version Maven property so that Scala 2.11 can be 
used
https://issues-test.apache.org/jira/browse/OOZIE-2571
OOZIE-2569  Adding yarn-site, core-site, hdfs-site and mapred-site into spark 
launcher 
https://issues-test.apache.org/jira/browse/OOZIE-2569
OOZIE-2568  SSH  action pretends to retry automaticly when it failed
https://issues-test.apache.org/jira/browse/OOZIE-2568
OOZIE-2564  Create new log4j config for unit tests so that logged messages 
contain threads
https://issues-test.apache.org/jira/browse/OOZIE-2564
OOZIE-2552  Update ActiveMQ version for security and other fixes
https://issues-test.apache.org/jira/browse/OOZIE-2552
OOZIE-2538  Update HttpClient versions to close security vulnerabilities
https://issues-test.apache.org/jira/browse/OOZIE-2538
OOZIE-2534  Versioned action libs (similar to how SharedLibs works)
https://issues-test.apache.org/jira/browse/OOZIE-2534
OOZIE-2530  Remove jline from parent and sharelib pig pom
https://issues-test.apache.org/jira/browse/OOZIE-2530
OOZIE-2528  Print out environment variables in LauncherMapper
https://issues-test.apache.org/jira/browse/OOZIE-2528
OOZIE-2521  Filter options are case sensitive for jobtye=bundle
https://issues-test.apache.org/jira/browse/OOZIE-2521
OOZIE-2518  Oozie is creating many files and directories in /tmp per day
https://issues-test.apache.org/jira/browse/OOZIE-2518
OOZIE-2517  Add support for startCreatedTime and endCreatedTime filters for 
coord and bundles
https://issues-test.apache.org/jira/browse/OOZIE-2517
OOZIE-2513  log.scan.duration should not be used for error and audit logs
https://issues-test.apache.org/jira/browse/OOZIE-2513
OOZIE-2512  ShareLibservice returns incorrect path for jar
https://issues-test.apache.org/jira/browse/OOZIE-2512
OOZIE-2508  Documentation change for Coord action rerun [OOZIE-1735]
https://issues-test.apache.org/jira/browse/OOZIE-2508
OOZIE-2501  ZK reentrant lock doesn't work for few cases
https://issues-test.apache.org/jira/browse/OOZIE-2501
OOZIE-2500  -DtestJarSimple option mentioned in minioozie doc does not work
https://issues-test.apache.org/jira/browse/OOZIE-2500
OOZIE-2499  map-reduce launcher does not need distributed files, archives 
except  jar of input/outputformat class
https://issues-test.apache.org/jira/browse/OOZIE-2499
OOZIE-2498  Oozie CallerId configuration for downstream components 
https://issues-test.apache.org/jira/browse/OOZIE-2498
OOZIE-2495  change action status from  ErrorType.NON_TRANSIENT to TRANSIENT 
when SSH action occurs AUTH_FAILED occasionally
https://issues-test.apache.org/jira/browse/OOZIE-2495
OOZIE-2491  oozie acl cannot specify group,it does`t work
https://issues-test.apache.org/jira/browse/OOZIE-2491

[jira] Subscription: Oozie Patch Available

2017-02-23 Thread jira 
Issue Subscription
Filter: Oozie Patch Available (94 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-2807  Oozie gets RM delegation token even for checking job status
https://issues.apache.org/jira/browse/OOZIE-2807
OOZIE-2806  Don't overwrite --files and --archives in Spark action
https://issues.apache.org/jira/browse/OOZIE-2806
OOZIE-2803  Mask passwords when printing out configs/args in MapReduceMain and 
SparkMain
https://issues.apache.org/jira/browse/OOZIE-2803
OOZIE-2802  Spark action failure on Spark 2.1.0 due to duplicate sharelibs
https://issues.apache.org/jira/browse/OOZIE-2802
OOZIE-2796  oozie.action.keep.action.dir not getting noticed
https://issues.apache.org/jira/browse/OOZIE-2796
OOZIE-2795  Create lib directory or symlink for Oozie CLI during packaging
https://issues.apache.org/jira/browse/OOZIE-2795
OOZIE-2791  ShareLib installation may fail on busy Hadoop clusters
https://issues.apache.org/jira/browse/OOZIE-2791
OOZIE-2785  Master is not compiling with hadoop-1 profile
https://issues.apache.org/jira/browse/OOZIE-2785
OOZIE-2784  Include WEEK as a parameter in the Coordinator Expression Language 
Evaulator
https://issues.apache.org/jira/browse/OOZIE-2784
OOZIE-2780  Upgrade hadoop-common version to 2.6.0
https://issues.apache.org/jira/browse/OOZIE-2780
OOZIE-2779  Mask Hive2 action Beeline JDBC password
https://issues.apache.org/jira/browse/OOZIE-2779
OOZIE-2769  Extend FS action to allow setrep on a file
https://issues.apache.org/jira/browse/OOZIE-2769
OOZIE-2753  Update Docs for Jetty Tomcat changes
https://issues.apache.org/jira/browse/OOZIE-2753
OOZIE-2751  LocalOozieClient is missing methods from OozieClient
https://issues.apache.org/jira/browse/OOZIE-2751
OOZIE-2750  Spelling errors in the log messages and exception messages
https://issues.apache.org/jira/browse/OOZIE-2750
OOZIE-2746  Several tests failure in TestV2ValidateServlet.java
https://issues.apache.org/jira/browse/OOZIE-2746
OOZIE-2739  Remove property expansion pattern from ShellMain's log4j properties 
content
https://issues.apache.org/jira/browse/OOZIE-2739
OOZIE-2736  Reduce the number of threads during test execution
https://issues.apache.org/jira/browse/OOZIE-2736
OOZIE-2733  change org.apache.hadoop.fs.permission.AccessControlException to 
org.apache.hadoop.security.AccessControlException
https://issues.apache.org/jira/browse/OOZIE-2733
OOZIE-2726  Flaky test due to daylight saving changes
https://issues.apache.org/jira/browse/OOZIE-2726
OOZIE-2718  Improve -dryrun for bundles
https://issues.apache.org/jira/browse/OOZIE-2718
OOZIE-2713  Provide an option to log out contents of the localized jar files
https://issues.apache.org/jira/browse/OOZIE-2713
OOZIE-2701  Oozie to support Multiple HCatalog URIs
https://issues.apache.org/jira/browse/OOZIE-2701
OOZIE-2694  Add logging for FsActionExecutor 
https://issues.apache.org/jira/browse/OOZIE-2694
OOZIE-2693  SimpleHCatDependencyCache.removeMissingDependency can throw NPE
https://issues.apache.org/jira/browse/OOZIE-2693
OOZIE-2692  Oozie job submit doesn't report error message to user if there is 
any issue with job conf
https://issues.apache.org/jira/browse/OOZIE-2692
OOZIE-2684  Bad database schema error for WF_ACTIONS table
https://issues.apache.org/jira/browse/OOZIE-2684
OOZIE-2681  fix javadoc to compile on JDK8 again
https://issues.apache.org/jira/browse/OOZIE-2681
OOZIE-2670  Upgrade Hbase to 1.2
https://issues.apache.org/jira/browse/OOZIE-2670
OOZIE-2668  Status update and recovery problems when coord action and its 
children not in sync
https://issues.apache.org/jira/browse/OOZIE-2668
OOZIE-2662  DB migration fails if DB is too big
https://issues.apache.org/jira/browse/OOZIE-2662
OOZIE-2652  Skip trash while deleting hive table partition
https://issues.apache.org/jira/browse/OOZIE-2652
OOZIE-2650  Retry coord start on database exceptions
https://issues.apache.org/jira/browse/OOZIE-2650
OOZIE-2644  Skip queuing Notification Commands when there's nothing to notify
https://issues.apache.org/jira/browse/OOZIE-2644
OOZIE-2641  rerunning an oozie coordinator job stucks on waiting
https://issues.apache.org/jira/browse/OOZIE-2641
OOZIE-2629  Get OozieShareLibCLI to perform a final rename to destPath when 
creating a Sharelib
https://issues.apache.org/jira/browse/OOZIE-2629
OOZIE-2618  Include hive-common in hive-beeline as compile component for 
hive-server2 (hive2 action)
https://issues.apache.org/jira/browse/OOZIE-2618
OOZIE-2608  Comma in oozie.service.JPAService.jdbc.password value results in 
authentication error

Build failed in Jenkins: oozie-trunk-precommit-build #3657

2017-02-23 Thread Apache Jenkins Server 
See 


--
[...truncated 1.44 MB...]
[INFO] Apache Oozie Examples .. SUCCESS [  6.010 s]
[INFO] Apache Oozie Share Lib Spark ... SUCCESS [ 22.987 s]
[INFO] Apache Oozie Share Lib . SUCCESS [01:02 min]
[INFO] Apache Oozie Docs .. SUCCESS [ 25.087 s]
[INFO] Apache Oozie WebApp  SUCCESS [01:09 min]
[INFO] Apache Oozie Tools . SUCCESS [  1.518 s]
[INFO] Apache Oozie MiniOozie . SUCCESS [  0.753 s]
[INFO] Apache Oozie Server  SUCCESS [  3.823 s]
[INFO] Apache Oozie Distro  SUCCESS [01:37 min]
[INFO] Apache Oozie ZooKeeper Security Tests .. SUCCESS [  3.222 s]
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 07:59 min
[INFO] Finished at: 2017-02-24T04:22:36+00:00
[INFO] Final Memory: 93M/1020M
[INFO] 

Reports
  Running test-patch task CLEAN
  Running test-patch task RAW_PATCH_ANALYSIS
  Running test-patch task RAT
  Running test-patch task JAVADOC
  Running test-patch task COMPILE
  Running test-patch task FINDBUGS_DIFF
[TRACE] Folder 
[
 does not exist, creating
[TRACE] XML file 
[
 does not exist, creating
[TRACE] Folder 
[
 does not exist, creating
[TRACE] XML file 
[
 does not exist, creating
[TRACE] Folder 
[
 does not exist, creating
[TRACE] XML file 
[
 does not exist, creating
[TRACE] Folder 
[
 does not exist, creating
[TRACE] XML file 
[
 does not exist, creating
[TRACE] Folder 
[
 does not exist, creating
[TRACE] XML file 
[
 does not exist, creating
[TRACE] Downloading FindBugs diff JAR from 
https://repo1.maven.org/maven2/me/andrz/findbugs/findbugs-diff/0.1.0/findbugs-diff-0.1.0-all.jar
[TRACE] FindBugs diff JAR downloaded
grep: 
:
 No such file or directory
[TRACE] FindBugs diff JAR checked, is safe to use
[TRACE] Performing FindBugs diffs
[TRACE] FindBugs diffs performed
[TRACE] Checking FindBugs diffs and creating reports
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [client].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/hive].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/sqoop].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[ERROR] There are [2] new bugs found below threshold in [sharelib/spark].
[DEBUG] You can find the FindBugs diff here (look for the red and orange ones): 
sharelib/spark/findbugs-new.html
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/streaming].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/hive2].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/oozie].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in

[jira] [Updated] (OOZIE-2808) a openjpa bug cause oozie start failed

2017-02-23 Thread zephyr.yi (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zephyr.yi updated OOZIE-2808:
-
Component/s: core

> a openjpa bug cause oozie start failed
> --
>
> Key: OOZIE-2808
> URL: https://issues.apache.org/jira/browse/OOZIE-2808
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 4.3.0
> Environment: a recompiled mysql version
> centos
>Reporter: zephyr.yi
>Priority: Minor
>
> with this bug describe
> https://issues.apache.org/jira/browse/OPENJPA-2687
> oozie server can't started, if openjpa donot slove this bug, can we have 
> other plan to fix this?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2807) Oozie gets RM delegation token even for checking job status

2017-02-23 Thread Satish Subhashrao Saley (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-2807:
---
Attachment: OOZIE-2807-1.patch

> Oozie gets RM delegation token even for checking job status
> ---
>
> Key: OOZIE-2807
> URL: https://issues.apache.org/jira/browse/OOZIE-2807
> Project: Oozie
>  Issue Type: Bug
>Reporter: Rohini Palaniswamy
>Assignee: Satish Subhashrao Saley
> Fix For: 5.0.0
>
> Attachments: OOZIE-2807-1.patch
>
>
> We had one user submitting way too many workflows with single hive query - 
> ~3600 workflows running concurrently. Surprisingly Oozie held up well without 
> issues.
> But [~daryn] from our hadoop team saw that the amount of delegation tokens 
> fetched by Oozie was very high compared to actual number of jobs submitted 
> and was stressing RM with the calls and also pushing it close to its memory 
> limits. This is because we are fetching the delegation token every time we 
> create a JobClient instead of only during job submission.
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java#L503-L519
> So for one job we fetch
> 1) 1 token during submission
> 2) 1 token every 5 minutes when we check status of job
> 3) 1 token after the job ends to retrieve status.
> 4) 1 token if we are killing the job.
> So for a job running for 11 minutes, we would have fetched the token 4 times. 
> May be more in other cases like mapreduce where we check for end of launcher 
> and child job.
> Only 1 out of the token (used in the job submission) will be cancelled after 
> job completes. Other tokens are kind of leaked and will only be cleaned up by 
> RM after the expiry period (24 hrs is default). This can make RM go out of 
> memory.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (OOZIE-2808) a openjpa bug cause oozie start failed

2017-02-23 Thread zephyr.yi (JIRA) 
zephyr.yi created OOZIE-2808:


 Summary: a openjpa bug cause oozie start failed
 Key: OOZIE-2808
 URL: https://issues.apache.org/jira/browse/OOZIE-2808
 Project: Oozie
  Issue Type: Bug
Affects Versions: 4.3.0
 Environment: a recompiled mysql version
centos
Reporter: zephyr.yi
Priority: Minor


with this bug describe
https://issues.apache.org/jira/browse/OPENJPA-2687

oozie server can't started, if openjpa donot slove this bug, can we have other 
plan to fix this?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain

2017-02-23 Thread Robert Kanter (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881715#comment-15881715
 ] 

Robert Kanter commented on OOZIE-2803:
--

Looks good overall.  A few trivial things:
- Can you look into the findbugs warning?
- When creating the new {{JobConf}} in {{MapReduceMain}}, I think it would be 
better to pass {{false}} to make sure it doesn't load any other properties if 
there's some site/default file on the classpath somehow.  All of the properties 
will be copied in there anyway.
{code:java}
JobConf maskedJobConf = new JobConf(false);
{code}
- Shouldn't {{PASSWORD_EXTRACTING_REGEX}} contain a reference to 
{{PASSWORD_KEY}} instead of hardcoding "pass" there?  I can't imagine we'd ever 
change {{PASSWORD_KEY}}, but that would be more future-proof.


> Mask passwords when printing out configs/args in MapReduceMain and SparkMain
> 
>
> Key: OOZIE-2803
> URL: https://issues.apache.org/jira/browse/OOZIE-2803
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch, 
> OOZIE-2803-003.patch, OOZIE-2803-004.patch
>
>
> Sometimes passwords are displayed in both MapReduce and Spark action.
> *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to 
> some Hadoop-specific config values, like {{mapred.child.env}}. This is easy 
> to fix because we already have a method {{logMasking()}} where you can define 
> a {{maskSet}} which contains a list of property keys to be masked.
> Note that this is not necessarily the perfect solution, since you can pass 
> multiple env. vars separated by a colon, and only the password specific parts 
> should be masked. But we need a working solution relatively quickly - later 
> we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right 
> now it only works with {{Map}}).
> *Spark*:  for Spark, we have to pass passwords like this:
> {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}}
> The Spark arguments are printed in {{SparkMain.run()}}. There is already a 
> code in {{LauncherMapper.printArgs()}} which deals with situations like this, 
> but it's not perfect because it only works if the args look something like 
> {{--password pwd123}}. So if a single arg contains a password, it doesn't 
> work, therefore we need a different approach here.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : oozie-trunk-find-patches-available #150755

2017-02-23 Thread Apache Jenkins Server 
See

Build failed in Jenkins: oozie-trunk-find-patches-available #150754

2017-02-23 Thread Apache Jenkins Server 
See 


--
Started by timer
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on H1 (Hadoop) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/oozie.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/oozie.git
 > git --version # timeout=10
 > git fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/oozie.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 77817e73588f665cbe43b0971cd93465d40711fb 
(refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 77817e73588f665cbe43b0971cd93465d40711fb
 > git rev-list 77817e73588f665cbe43b0971cd93465d40711fb # timeout=10
[oozie-trunk-find-patches-available] $ /bin/bash 
/tmp/hudson7170102141718465019.sh
mkdir: cannot create directory 
':
 File exists
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100  279k0  279k0 0   304k  0 --:--:-- --:--:-- --:--:--  
304k100  551k0  551k0 0   397k  0 --:--:--  0:00:01 --:--:--  
397k
curl: (18) transfer closed with outstanding read data remaining
Could not retrieve available patches from JIRA
Build step 'Execute shell' marked build as failure