[jira] Subscription: Oozie Patch Available
Issue Subscription Filter: Oozie Patch Available (95 issues) Subscriber: ooziedaily Key Summary OOZIE-3579 [docs] Fix typos in coordinator documentation https://issues.apache.org/jira/browse/OOZIE-3579 OOZIE-3574 JavaAction create incorrect fileSystem instance in addActionLibs method https://issues.apache.org/jira/browse/OOZIE-3574 OOZIE-3569 SSH Action should add checking success file https://issues.apache.org/jira/browse/OOZIE-3569 OOZIE-3568 Have large amount of log information “WARN messages [main] openjpa.MetaData” in jetty.log need to clean https://issues.apache.org/jira/browse/OOZIE-3568 OOZIE-3567 Oozie ShellAction should support absolute bash file path https://issues.apache.org/jira/browse/OOZIE-3567 OOZIE-3560 IDEA shows have some error in index.jsp https://issues.apache.org/jira/browse/OOZIE-3560 OOZIE-3482 Fix bug in CoordSubmitXCommand#validateCoordinatorJob https://issues.apache.org/jira/browse/OOZIE-3482 OOZIE-3480 Add windowactionstatus metrics in DBLiteWorkflowStoreService https://issues.apache.org/jira/browse/OOZIE-3480 OOZIE-3461 CoordMaterializeTriggerService code cleanup https://issues.apache.org/jira/browse/OOZIE-3461 OOZIE-3449 Make spark-2 as the default profile https://issues.apache.org/jira/browse/OOZIE-3449 OOZIE-3447 Run test case in local : It shows oozie-hsqldb-orm.xml exception https://issues.apache.org/jira/browse/OOZIE-3447 OOZIE-3418 Upgrade to Guava 27 https://issues.apache.org/jira/browse/OOZIE-3418 OOZIE-3404 The env variable of SPARK_HOME needs to be set when running pySpark https://issues.apache.org/jira/browse/OOZIE-3404 OOZIE-3375 Can't use empty in coordinator https://issues.apache.org/jira/browse/OOZIE-3375 OOZIE-3367 Using && in EL expressions in oozie bundle.xml files generates parse errors https://issues.apache.org/jira/browse/OOZIE-3367 OOZIE-3366 Update workflow status and subworkflow status on suspend command https://issues.apache.org/jira/browse/OOZIE-3366 OOZIE-3364 Rerunning Oozie bundle jobs starts the coordinators in indeterminate order https://issues.apache.org/jira/browse/OOZIE-3364 OOZIE-3362 When killed, SSH action should kill the spawned processes on target host https://issues.apache.org/jira/browse/OOZIE-3362 OOZIE-3335 Cleanup parseFilter methods https://issues.apache.org/jira/browse/OOZIE-3335 OOZIE-3328 Create Hive compatibility action executor to run hive actions using beeline https://issues.apache.org/jira/browse/OOZIE-3328 OOZIE-3319 Log SSH action callback error output https://issues.apache.org/jira/browse/OOZIE-3319 OOZIE-3301 Update NOTICE file https://issues.apache.org/jira/browse/OOZIE-3301 OOZIE-3274 Remove slf4j https://issues.apache.org/jira/browse/OOZIE-3274 OOZIE-3266 Coord action rerun support RERUN_SKIP_NODES option https://issues.apache.org/jira/browse/OOZIE-3266 OOZIE-3256 refactor OozieCLI class https://issues.apache.org/jira/browse/OOZIE-3256 OOZIE-3254 [coordinator] LAST_ONLY and NONE execution modes: possible OutOfMemoryError when there are too many coordinator actions to materialize https://issues.apache.org/jira/browse/OOZIE-3254 OOZIE-3199 Let system property restriction configurable https://issues.apache.org/jira/browse/OOZIE-3199 OOZIE-3196 Authorization: restrict world readability by user https://issues.apache.org/jira/browse/OOZIE-3196 OOZIE-3170 Oozie Diagnostic Bundle tool fails with NPE due to missing service class https://issues.apache.org/jira/browse/OOZIE-3170 OOZIE-3137 Add support for log4j2 in HiveMain https://issues.apache.org/jira/browse/OOZIE-3137 OOZIE-3135 Configure log4j2 in SqoopMain https://issues.apache.org/jira/browse/OOZIE-3135 OOZIE-3091 Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: org/apache/avro/mapred/AvroWrapper" https://issues.apache.org/jira/browse/OOZIE-3091 OOZIE-3071 Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than Spark 2.2.0 https://issues.apache.org/jira/browse/OOZIE-3071 OOZIE-3066 Possibility to set retry-max globally https://issues.apache.org/jira/browse/OOZIE-3066 OOZIE-3063 Sanitizing variables that are part of openjpa.ConnectionProperties https://issues.apache.org/jira/browse/OOZIE-3063 OOZIE-3062 Set HADOOP_CONF_DIR for spark action https://issues.apache.org/jira/browse/OOZIE-3062 OOZIE-2952 Fix Findbugs warnings in oozie-sharelib-oozie https://issues.apache.org/jira/browse/OOZIE-2952 OOZIE-2834 ParameterVerifier logging non-useful warning for workflow definition https://issues.apache.org/jira/browse/OOZIE-2834 OOZIE-2812
[jira] [Commented] (OOZIE-3066) Possibility to set retry-max globally
[
https://issues.apache.org/jira/browse/OOZIE-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011467#comment-17011467
]
Hadoop QA commented on OOZIE-3066:
--
PreCommit-OOZIE-Build started
> Possibility to set retry-max globally
> -
>
> Key: OOZIE-3066
> URL: https://issues.apache.org/jira/browse/OOZIE-3066
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: trunk
>Reporter: Gézapeti
>Assignee: Qs Bao
>Priority: Major
> Attachments: OOZIE-3066-001.patch, OOZIE-3066-002.patch,
> OOZIE-3066-003.patch
>
>
> Currently there is no way to set the equivalent of {{retry-max}} globally to
> apply to every action.
> It should be possible to set a default value to that to make workflow
> execution more reliable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3066) Possibility to set retry-max globally
[
https://issues.apache.org/jira/browse/OOZIE-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011458#comment-17011458
]
Qs Bao commented on OOZIE-3066:
---
Third patch fix long lines.
> Possibility to set retry-max globally
> -
>
> Key: OOZIE-3066
> URL: https://issues.apache.org/jira/browse/OOZIE-3066
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: trunk
>Reporter: Gézapeti
>Assignee: Qs Bao
>Priority: Major
> Attachments: OOZIE-3066-001.patch, OOZIE-3066-002.patch,
> OOZIE-3066-003.patch
>
>
> Currently there is no way to set the equivalent of {{retry-max}} globally to
> apply to every action.
> It should be possible to set a default value to that to make workflow
> execution more reliable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3066) Possibility to set retry-max globally
[
https://issues.apache.org/jira/browse/OOZIE-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qs Bao updated OOZIE-3066:
--
Attachment: OOZIE-3066-003.patch
> Possibility to set retry-max globally
> -
>
> Key: OOZIE-3066
> URL: https://issues.apache.org/jira/browse/OOZIE-3066
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: trunk
>Reporter: Gézapeti
>Assignee: Qs Bao
>Priority: Major
> Attachments: OOZIE-3066-001.patch, OOZIE-3066-002.patch,
> OOZIE-3066-003.patch
>
>
> Currently there is no way to set the equivalent of {{retry-max}} globally to
> apply to every action.
> It should be possible to set a default value to that to make workflow
> execution more reliable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3066) Possibility to set retry-max globally
[
https://issues.apache.org/jira/browse/OOZIE-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010743#comment-17010743
]
Hadoop QA commented on OOZIE-3066:
--
Testing JIRA OOZIE-3066
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:red}-1{color} the patch contains 5 line(s) longer than 132
characters
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [23] new bugs found below threshold in total
that must be fixed.
.{color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:red}-1{color} There are [7] new bugs found below threshold in
[core] that must be fixed, listing only the first [5] ones.
.You can find the SpotBugs diff here (look for the red and orange ones):
core/findbugs-new.html
.The top [5] most important SpotBugs errors are:
.At BulkJPAExecutor.java:[line 206]: This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection
.At BulkJPAExecutor.java:[line 176]: At BulkJPAExecutor.java:[line 175]
.At BulkJPAExecutor.java:[line 205]: At BulkJPAExecutor.java:[line 199]
.This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection: At BulkJPAExecutor.java:[line 206]
.At BulkJPAExecutor.java:[line 111]: At BulkJPAExecutor.java:[line 127]
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in [sharelib/git].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:red}-1{color} There are [1] new bugs found below threshold in
[sharelib/oozie] that must be fixed.
.You can find the SpotBugs diff here (look for the red and orange ones):
sharelib/oozie/findbugs-new.html
.The most important SpotBugs errors are:
.At ShellMain.java:[line 93]: This usage of
java/lang/ProcessBuilder.init(Ljava/util/List;)V can be vulnerable to
Command Injection
.At ShellMain.java:[line 91]: At ShellMain.java:[line 90]
.At ShellMain.java:[line 92]
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:red}-1{color} There are [15] new bugs found below threshold in
[tools] that must be fixed, listing only the first [5] ones.
.You can find the SpotBugs diff here (look for the red and orange ones):
tools/findbugs-new.html
.The top [5] most important SpotBugs errors are:
.At OozieDBCLI.java:[line 584]: This use of
java/sql/Statement.executeUpdate(Ljava/lang/String;)I can be vulnerable to SQL
injection
.At OozieDBCLI.java:[line 574]: At OozieDBCLI.java:[line 573]
.At OozieDBCLI.java:[line 577]: At OozieDBCLI.java:[line 575]
.At OozieDBCLI.java:[line 579]: At OozieDBCLI.java:[line 578]
.At OozieDBCLI.java:[line 584]: At OozieDBCLI.java:[line 581]
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [webapp].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 3201
.{color:orange}Tests failed at first run:{color}
Failed: OOZIE-3066 PreCommit Build #1281
Jira: https://issues.apache.org/jira/browse/OOZIE-3066 Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/1281/ ### ## LAST 100 LINES OF THE CONSOLE ### [...truncated 2.16 MB...] Testing JIRA OOZIE-3066 Cleaning local git workspace +1 PATCH_APPLIES +1 CLEAN -1 RAW_PATCH_ANALYSIS +1 the patch does not introduce any @author tags +1 the patch does not introduce any tabs +1 the patch does not introduce any trailing spaces +1 the patch does not introduce any star imports -1 the patch contains 5 line(s) longer than 132 characters +1 the patch adds/modifies 1 testcase(s) +1 RAT +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC +1 Javadoc generation succeeded with the patch +1 the patch does not seem to introduce new Javadoc warning(s) +1 COMPILE +1 HEAD compiles +1 patch compiles +1 the patch does not seem to introduce new javac warnings -1 There are [23] new bugs found below threshold in total that must be fixed. +1 There are no new bugs found in [fluent-job/fluent-job-api]. +1 There are no new bugs found in [docs]. -1 There are [7] new bugs found below threshold in [core] that must be fixed, listing only the first [5] ones. You can find the SpotBugs diff here (look for the red and orange ones): core/findbugs-new.html The top [5] most important SpotBugs errors are: At BulkJPAExecutor.java:[line 206]: This use of javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query; can be vulnerable to SQL/JPQL injection At BulkJPAExecutor.java:[line 176]: At BulkJPAExecutor.java:[line 175] At BulkJPAExecutor.java:[line 205]: At BulkJPAExecutor.java:[line 199] This use of javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query; can be vulnerable to SQL/JPQL injection: At BulkJPAExecutor.java:[line 206] At BulkJPAExecutor.java:[line 111]: At BulkJPAExecutor.java:[line 127] +1 There are no new bugs found in [sharelib/spark]. +1 There are no new bugs found in [sharelib/git]. +1 There are no new bugs found in [sharelib/sqoop]. +1 There are no new bugs found in [sharelib/hive2]. +1 There are no new bugs found in [sharelib/streaming]. +1 There are no new bugs found in [sharelib/pig]. -1 There are [1] new bugs found below threshold in [sharelib/oozie] that must be fixed. You can find the SpotBugs diff here (look for the red and orange ones): sharelib/oozie/findbugs-new.html The most important SpotBugs errors are: At ShellMain.java:[line 93]: This usage of java/lang/ProcessBuilder.init(Ljava/util/List;)V can be vulnerable to Command Injection At ShellMain.java:[line 91]: At ShellMain.java:[line 90] At ShellMain.java:[line 92] +1 There are no new bugs found in [sharelib/hive]. +1 There are no new bugs found in [sharelib/hcatalog]. +1 There are no new bugs found in [sharelib/distcp]. -1 There are [15] new bugs found below threshold in [tools] that must be fixed, listing only the first [5] ones. You can find the SpotBugs diff here (look for the red and orange ones): tools/findbugs-new.html The top [5] most important SpotBugs errors are: At OozieDBCLI.java:[line 584]: This use of java/sql/Statement.executeUpdate(Ljava/lang/String;)I can be vulnerable to SQL injection At OozieDBCLI.java:[line 574]: At OozieDBCLI.java:[line 573] At OozieDBCLI.java:[line 577]: At OozieDBCLI.java:[line 575] At OozieDBCLI.java:[line 579]: At OozieDBCLI.java:[line 578] At OozieDBCLI.java:[line 584]: At OozieDBCLI.java:[line 581] +1 There are no new bugs found in [server]. +1 There are no new bugs found in [client]. +1 There are no new bugs found in [examples]. +1 There are no new bugs found in [webapp]. +1 BACKWARDS_COMPATIBILITY +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations +1 the patch does not modify JPA files +1 TESTS Tests run: 3201 Tests failed at first run: TestBlockingInputStream#testFastWritingBlockingInputStream TestBlockingInputStream#testLimitedWritingBlockingInputStream TestPriorityDelayQueue#testPeek For the complete list of flaky tests, see TEST-SUMMARY-FULL files. +1 DISTRO +1 distro tarball builds with the patch +1 MODERNIZER -1 Overall result, please check the reported -1(s) The full output of the test-patch run is available at https://builds.apache.org/job/PreCommit-OOZIE-Build/1281/ Adding comment to JIRA % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0100 54230 0 100 5423 0 3589
[jira] [Commented] (OOZIE-3066) Possibility to set retry-max globally
[
https://issues.apache.org/jira/browse/OOZIE-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010660#comment-17010660
]
Hadoop QA commented on OOZIE-3066:
--
PreCommit-OOZIE-Build started
> Possibility to set retry-max globally
> -
>
> Key: OOZIE-3066
> URL: https://issues.apache.org/jira/browse/OOZIE-3066
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: trunk
>Reporter: Gézapeti
>Assignee: Qs Bao
>Priority: Major
> Attachments: OOZIE-3066-001.patch, OOZIE-3066-002.patch
>
>
> Currently there is no way to set the equivalent of {{retry-max}} globally to
> apply to every action.
> It should be possible to set a default value to that to make workflow
> execution more reliable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3066) Possibility to set retry-max globally
[
https://issues.apache.org/jira/browse/OOZIE-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010650#comment-17010650
]
Qs Bao commented on OOZIE-3066:
---
[~asalamon74] Thanks for the reviewing, second patch submitted.
> Possibility to set retry-max globally
> -
>
> Key: OOZIE-3066
> URL: https://issues.apache.org/jira/browse/OOZIE-3066
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: trunk
>Reporter: Gézapeti
>Assignee: Qs Bao
>Priority: Major
> Attachments: OOZIE-3066-001.patch, OOZIE-3066-002.patch
>
>
> Currently there is no way to set the equivalent of {{retry-max}} globally to
> apply to every action.
> It should be possible to set a default value to that to make workflow
> execution more reliable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3066) Possibility to set retry-max globally
[
https://issues.apache.org/jira/browse/OOZIE-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qs Bao updated OOZIE-3066:
--
Attachment: OOZIE-3066-002.patch
> Possibility to set retry-max globally
> -
>
> Key: OOZIE-3066
> URL: https://issues.apache.org/jira/browse/OOZIE-3066
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: trunk
>Reporter: Gézapeti
>Assignee: Qs Bao
>Priority: Major
> Attachments: OOZIE-3066-001.patch, OOZIE-3066-002.patch
>
>
> Currently there is no way to set the equivalent of {{retry-max}} globally to
> apply to every action.
> It should be possible to set a default value to that to make workflow
> execution more reliable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3578) MapReduce counters cannot be used over 120
[
https://issues.apache.org/jira/browse/OOZIE-3578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010649#comment-17010649
]
ASF subversion and git services commented on OOZIE-3578:
Commit f0af3a01c68e373cba3122ac7a005b987d84c729 in oozie's branch
refs/heads/master from Gezapeti Cseh
[ https://gitbox.apache.org/repos/asf?p=oozie.git;h=f0af3a0 ]
OOZIE-3578 MapReduce counters cannot be used over 120 (dionusos via pbacsko,
gezapeti)
> MapReduce counters cannot be used over 120
> --
>
> Key: OOZIE-3578
> URL: https://issues.apache.org/jira/browse/OOZIE-3578
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.1.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Attachments: OOZIE-3578-001.patch, OOZIE-3578-002.patch,
> OOZIE-3578-003.patch
>
>
> When we create a mapreduce action which then creates more than 120 counters
> then the following exception is thrown:
> {noformat}
> org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:101)
> org.apache.hadoop.mapreduce.counters.Limits.incrCounters(Limits.java:108)
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:78)
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:95)
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounterImpl(AbstractCounterGroup.java:123)
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:113)
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:130)
> org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:155)
> org.apache.hadoop.mapreduce.TypeConverter.fromYarn(TypeConverter.java:264)
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:383)
> org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:859)
> org.apache.hadoop.mapreduce.Job$8.run(Job.java:820)
> org.apache.hadoop.mapreduce.Job$8.run(Job.java:817)
> java.security.AccessController.doPrivileged(Native Method)
> javax.security.auth.Subject.doAs(Subject.java:422)
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
> org.apache.hadoop.mapreduce.Job.getCounters(Job.java:817)
> org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:379)
> org.apache.oozie.action.hadoop.MapReduceActionExecutor.end(MapReduceActionExecutor.java:252)
> org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:183)
> org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:62)
> org.apache.oozie.command.XCommand.call(XCommand.java:291)
> org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:244)
> org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:56)
> org.apache.oozie.command.XCommand.call(XCommand.java:291)
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:210)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> {noformat}
> It turned out if we use Oozie with Hadoop 3 the MR class called {{Limits}} is
> not initialised properly but with default values:
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/Limits.java#L40
> If we set the "mapreduce.job.counters.max" to 500 in mapred-site.xml or in
> core-site.xml has no positive effect.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3578) MapReduce counters cannot be used over 120
[
https://issues.apache.org/jira/browse/OOZIE-3578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010648#comment-17010648
]
Gézapeti commented on OOZIE-3578:
-
+1
committed to master!
Thanks for the contribution [~dionusos]!
> MapReduce counters cannot be used over 120
> --
>
> Key: OOZIE-3578
> URL: https://issues.apache.org/jira/browse/OOZIE-3578
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.1.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Attachments: OOZIE-3578-001.patch, OOZIE-3578-002.patch,
> OOZIE-3578-003.patch
>
>
> When we create a mapreduce action which then creates more than 120 counters
> then the following exception is thrown:
> {noformat}
> org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:101)
> org.apache.hadoop.mapreduce.counters.Limits.incrCounters(Limits.java:108)
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:78)
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:95)
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounterImpl(AbstractCounterGroup.java:123)
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:113)
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:130)
> org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:155)
> org.apache.hadoop.mapreduce.TypeConverter.fromYarn(TypeConverter.java:264)
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:383)
> org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:859)
> org.apache.hadoop.mapreduce.Job$8.run(Job.java:820)
> org.apache.hadoop.mapreduce.Job$8.run(Job.java:817)
> java.security.AccessController.doPrivileged(Native Method)
> javax.security.auth.Subject.doAs(Subject.java:422)
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
> org.apache.hadoop.mapreduce.Job.getCounters(Job.java:817)
> org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:379)
> org.apache.oozie.action.hadoop.MapReduceActionExecutor.end(MapReduceActionExecutor.java:252)
> org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:183)
> org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:62)
> org.apache.oozie.command.XCommand.call(XCommand.java:291)
> org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:244)
> org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:56)
> org.apache.oozie.command.XCommand.call(XCommand.java:291)
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:210)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> {noformat}
> It turned out if we use Oozie with Hadoop 3 the MR class called {{Limits}} is
> not initialised properly but with default values:
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/Limits.java#L40
> If we set the "mapreduce.job.counters.max" to 500 in mapred-site.xml or in
> core-site.xml has no positive effect.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Failed: OOZIE-3578 PreCommit Build #1280
Jira: https://issues.apache.org/jira/browse/OOZIE-3578 Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/1280/ ### ## LAST 100 LINES OF THE CONSOLE ### [...truncated 2.15 MB...] [TRACE] grep -c '.*ERROR.*Prefer' /home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/reports/MODERNIZER-clean.txt [TRACE] Modernizer errors before patch: 51 [TRACE] grep -c '.*ERROR.*Prefer' /home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/reports/MODERNIZER-patch.txt [TRACE] Modernizer errors after patch: 51 Testing JIRA OOZIE-3578 Cleaning local git workspace +1 PATCH_APPLIES +1 CLEAN +1 RAW_PATCH_ANALYSIS +1 the patch does not introduce any @author tags +1 the patch does not introduce any tabs +1 the patch does not introduce any trailing spaces +1 the patch does not introduce any star imports +1 the patch does not introduce any line longer than 132 +1 the patch adds/modifies 1 testcase(s) +1 RAT +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC +1 Javadoc generation succeeded with the patch +1 the patch does not seem to introduce new Javadoc warning(s) +1 COMPILE +1 HEAD compiles +1 patch compiles +1 the patch does not seem to introduce new javac warnings -1 There are [20] new bugs found below threshold in total that must be fixed. -1 There are [5] new bugs found below threshold in [core] that must be fixed. You can find the SpotBugs diff here (look for the red and orange ones): core/findbugs-new.html The most important SpotBugs errors are: At BulkJPAExecutor.java:[line 206]: This use of javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query; can be vulnerable to SQL/JPQL injection At BulkJPAExecutor.java:[line 176]: At BulkJPAExecutor.java:[line 175] At BulkJPAExecutor.java:[line 205]: At BulkJPAExecutor.java:[line 199] This use of javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query; can be vulnerable to SQL/JPQL injection: At BulkJPAExecutor.java:[line 206] At CoordJobGetActionsSubsetJPAExecutor.java:[line 76]: At CoordJobGetActionsSubsetJPAExecutor.java:[line 111] -1 There are [15] new bugs found below threshold in [tools] that must be fixed, listing only the first [5] ones. You can find the SpotBugs diff here (look for the red and orange ones): tools/findbugs-new.html The top [5] most important SpotBugs errors are: At OozieDBCLI.java:[line 584]: This use of java/sql/Statement.executeUpdate(Ljava/lang/String;)I can be vulnerable to SQL injection At OozieDBCLI.java:[line 574]: At OozieDBCLI.java:[line 573] At OozieDBCLI.java:[line 577]: At OozieDBCLI.java:[line 575] At OozieDBCLI.java:[line 579]: At OozieDBCLI.java:[line 578] At OozieDBCLI.java:[line 584]: At OozieDBCLI.java:[line 581] +1 There are no new bugs found in [docs]. +1 There are no new bugs found in [sharelib/streaming]. +1 There are no new bugs found in [sharelib/spark]. +1 There are no new bugs found in [sharelib/hive2]. +1 There are no new bugs found in [sharelib/hcatalog]. +1 There are no new bugs found in [sharelib/distcp]. +1 There are no new bugs found in [sharelib/hive]. +1 There are no new bugs found in [sharelib/oozie]. +1 There are no new bugs found in [sharelib/sqoop]. +1 There are no new bugs found in [sharelib/pig]. +1 There are no new bugs found in [sharelib/git]. +1 There are no new bugs found in [webapp]. +1 There are no new bugs found in [client]. +1 There are no new bugs found in [examples]. +1 There are no new bugs found in [fluent-job/fluent-job-api]. 0 There are [4] new bugs found in [server] that would be nice to have fixed. You can find the SpotBugs diff here: server/findbugs-new.html +1 BACKWARDS_COMPATIBILITY +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations +1 the patch does not modify JPA files +1 TESTS Tests run: 3200 Tests failed at first run: TestBlockingInputStream#testFastWritingBlockingInputStream TestPriorityDelayQueue#testPeek For the complete list of flaky tests, see TEST-SUMMARY-FULL files. +1 DISTRO +1 distro tarball builds with the patch +1 MODERNIZER -1 Overall result, please check the reported -1(s) The full output of the test-patch run is available at https://builds.apache.org/job/PreCommit-OOZIE-Build/1280/ Adding comment to JIRA % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0100
[jira] [Commented] (OOZIE-3578) MapReduce counters cannot be used over 120
[
https://issues.apache.org/jira/browse/OOZIE-3578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010636#comment-17010636
]
Hadoop QA commented on OOZIE-3578:
--
Testing JIRA OOZIE-3578
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [20] new bugs found below threshold in total
that must be fixed.
.{color:red}-1{color} There are [5] new bugs found below threshold in
[core] that must be fixed.
.You can find the SpotBugs diff here (look for the red and orange ones):
core/findbugs-new.html
.The most important SpotBugs errors are:
.At BulkJPAExecutor.java:[line 206]: This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection
.At BulkJPAExecutor.java:[line 176]: At BulkJPAExecutor.java:[line 175]
.At BulkJPAExecutor.java:[line 205]: At BulkJPAExecutor.java:[line 199]
.This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection: At BulkJPAExecutor.java:[line 206]
.At CoordJobGetActionsSubsetJPAExecutor.java:[line 76]: At
CoordJobGetActionsSubsetJPAExecutor.java:[line 111]
.{color:red}-1{color} There are [15] new bugs found below threshold in
[tools] that must be fixed, listing only the first [5] ones.
.You can find the SpotBugs diff here (look for the red and orange ones):
tools/findbugs-new.html
.The top [5] most important SpotBugs errors are:
.At OozieDBCLI.java:[line 584]: This use of
java/sql/Statement.executeUpdate(Ljava/lang/String;)I can be vulnerable to SQL
injection
.At OozieDBCLI.java:[line 574]: At OozieDBCLI.java:[line 573]
.At OozieDBCLI.java:[line 577]: At OozieDBCLI.java:[line 575]
.At OozieDBCLI.java:[line 579]: At OozieDBCLI.java:[line 578]
.At OozieDBCLI.java:[line 584]: At OozieDBCLI.java:[line 581]
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/git].
.{color:green}+1{color} There are no new bugs found in [webapp].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
.{color:orange}0{color} There are [4] new bugs found in [server] that would
be nice to have fixed.
.You can find the SpotBugs diff here: server/findbugs-new.html
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 3200
.{color:orange}Tests failed at first run:{color}
TestBlockingInputStream#testFastWritingBlockingInputStream
TestPriorityDelayQueue#testPeek
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:green}+1 MODERNIZER{color}
[jira] [Commented] (OOZIE-3578) MapReduce counters cannot be used over 120
[
https://issues.apache.org/jira/browse/OOZIE-3578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010556#comment-17010556
]
Hadoop QA commented on OOZIE-3578:
--
PreCommit-OOZIE-Build started
> MapReduce counters cannot be used over 120
> --
>
> Key: OOZIE-3578
> URL: https://issues.apache.org/jira/browse/OOZIE-3578
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.1.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Attachments: OOZIE-3578-001.patch, OOZIE-3578-002.patch,
> OOZIE-3578-003.patch
>
>
> When we create a mapreduce action which then creates more than 120 counters
> then the following exception is thrown:
> {noformat}
> org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:101)
> org.apache.hadoop.mapreduce.counters.Limits.incrCounters(Limits.java:108)
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:78)
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:95)
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounterImpl(AbstractCounterGroup.java:123)
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:113)
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:130)
> org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:155)
> org.apache.hadoop.mapreduce.TypeConverter.fromYarn(TypeConverter.java:264)
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:383)
> org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:859)
> org.apache.hadoop.mapreduce.Job$8.run(Job.java:820)
> org.apache.hadoop.mapreduce.Job$8.run(Job.java:817)
> java.security.AccessController.doPrivileged(Native Method)
> javax.security.auth.Subject.doAs(Subject.java:422)
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
> org.apache.hadoop.mapreduce.Job.getCounters(Job.java:817)
> org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:379)
> org.apache.oozie.action.hadoop.MapReduceActionExecutor.end(MapReduceActionExecutor.java:252)
> org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:183)
> org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:62)
> org.apache.oozie.command.XCommand.call(XCommand.java:291)
> org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:244)
> org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:56)
> org.apache.oozie.command.XCommand.call(XCommand.java:291)
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:210)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> {noformat}
> It turned out if we use Oozie with Hadoop 3 the MR class called {{Limits}} is
> not initialised properly but with default values:
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/Limits.java#L40
> If we set the "mapreduce.job.counters.max" to 500 in mapred-site.xml or in
> core-site.xml has no positive effect.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3578) MapReduce counters cannot be used over 120
[
https://issues.apache.org/jira/browse/OOZIE-3578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo updated OOZIE-3578:
--
Attachment: OOZIE-3578-003.patch
> MapReduce counters cannot be used over 120
> --
>
> Key: OOZIE-3578
> URL: https://issues.apache.org/jira/browse/OOZIE-3578
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.1.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Attachments: OOZIE-3578-001.patch, OOZIE-3578-002.patch,
> OOZIE-3578-003.patch
>
>
> When we create a mapreduce action which then creates more than 120 counters
> then the following exception is thrown:
> {noformat}
> org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:101)
> org.apache.hadoop.mapreduce.counters.Limits.incrCounters(Limits.java:108)
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:78)
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:95)
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounterImpl(AbstractCounterGroup.java:123)
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:113)
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:130)
> org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:155)
> org.apache.hadoop.mapreduce.TypeConverter.fromYarn(TypeConverter.java:264)
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:383)
> org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:859)
> org.apache.hadoop.mapreduce.Job$8.run(Job.java:820)
> org.apache.hadoop.mapreduce.Job$8.run(Job.java:817)
> java.security.AccessController.doPrivileged(Native Method)
> javax.security.auth.Subject.doAs(Subject.java:422)
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
> org.apache.hadoop.mapreduce.Job.getCounters(Job.java:817)
> org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:379)
> org.apache.oozie.action.hadoop.MapReduceActionExecutor.end(MapReduceActionExecutor.java:252)
> org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:183)
> org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:62)
> org.apache.oozie.command.XCommand.call(XCommand.java:291)
> org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:244)
> org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:56)
> org.apache.oozie.command.XCommand.call(XCommand.java:291)
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:210)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> {noformat}
> It turned out if we use Oozie with Hadoop 3 the MR class called {{Limits}} is
> not initialised properly but with default values:
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/Limits.java#L40
> If we set the "mapreduce.job.counters.max" to 500 in mapred-site.xml or in
> core-site.xml has no positive effect.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3578) MapReduce counters cannot be used over 120
[
https://issues.apache.org/jira/browse/OOZIE-3578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010531#comment-17010531
]
Peter Bacsko commented on OOZIE-3578:
-
Thanks [~dionusos] IMO the patch is good.
Just a couple of nits:
# Instead of hardcoding ""mapreduce.job.counters.max", you can use
{{MRJobConfig.COUNTERS_MAX_KEY}}
# Instead of {{get("mapreduce.job.counters.max")}}, you can use {{getInt()}}
so you don't need {{Integer.parseInt()}}
> MapReduce counters cannot be used over 120
> --
>
> Key: OOZIE-3578
> URL: https://issues.apache.org/jira/browse/OOZIE-3578
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.1.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Attachments: OOZIE-3578-001.patch, OOZIE-3578-002.patch
>
>
> When we create a mapreduce action which then creates more than 120 counters
> then the following exception is thrown:
> {noformat}
> org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:101)
> org.apache.hadoop.mapreduce.counters.Limits.incrCounters(Limits.java:108)
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:78)
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:95)
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounterImpl(AbstractCounterGroup.java:123)
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:113)
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:130)
> org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:155)
> org.apache.hadoop.mapreduce.TypeConverter.fromYarn(TypeConverter.java:264)
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:383)
> org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:859)
> org.apache.hadoop.mapreduce.Job$8.run(Job.java:820)
> org.apache.hadoop.mapreduce.Job$8.run(Job.java:817)
> java.security.AccessController.doPrivileged(Native Method)
> javax.security.auth.Subject.doAs(Subject.java:422)
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
> org.apache.hadoop.mapreduce.Job.getCounters(Job.java:817)
> org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:379)
> org.apache.oozie.action.hadoop.MapReduceActionExecutor.end(MapReduceActionExecutor.java:252)
> org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:183)
> org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:62)
> org.apache.oozie.command.XCommand.call(XCommand.java:291)
> org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:244)
> org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:56)
> org.apache.oozie.command.XCommand.call(XCommand.java:291)
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:210)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> {noformat}
> It turned out if we use Oozie with Hadoop 3 the MR class called {{Limits}} is
> not initialised properly but with default values:
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/Limits.java#L40
> If we set the "mapreduce.job.counters.max" to 500 in mapred-site.xml or in
> core-site.xml has no positive effect.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3569) SSH Action should add checking success file
[
https://issues.apache.org/jira/browse/OOZIE-3569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010513#comment-17010513
]
Junfan Zhang commented on OOZIE-3569:
-
[~asalamon74] OK, i will extract error file checking to a new method.
In the production environment, we are gradually applying this patch, but this
change is harmful. If the SSH action was run before applying this patch, so the
success file is not generated, then after the action ends, the success file
will be checked, finally the action will be misjudged as a failure.
So I first added the touch success file, but the success file was not verified
at the end of the action. After one week, I applying this patch completely with
the checking success file strategy.
After 2 weeks of stable running, I will reply here.
> SSH Action should add checking success file
> ---
>
> Key: OOZIE-3569
> URL: https://issues.apache.org/jira/browse/OOZIE-3569
> Project: Oozie
> Issue Type: Bug
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Attachments: OOZIE-3569-v1.patch
>
>
> *Phenomenon*
> Currently, {{SSH Action}} checking operation are as following:
> Firstly, check operation is to check {{Oozie}} ppid. When pgid does not
> exist, check whether there is an error file. If the error file does not
> exist, {{Oozie}} will set action status {{OK}}
> However, when {{Oozie}} pgid is killed externally, this action will be
> incorrectly determined to be successful.
> *Solution*
> In ssh-wrapper.sh, when command execution is OK, {{Oozie}} should touch a
> success empty file like touching error file.
> In {{SshActionExecutor}} check method, Oozie should add checking the success
> file existence.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3066) Possibility to set retry-max globally
[
https://issues.apache.org/jira/browse/OOZIE-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010494#comment-17010494
]
Andras Salamon commented on OOZIE-3066:
---
Thanks for the quick patch [~qsbao] I like the approach, I only have a few
suggestions:
* It's a very good idea to check the default and the max and return max if
default>max. I think we should also print out a warning message in this case.
* The default 0 is hardwired twice in the code. Once in {{oozie-default.xml}}
and once in the {{ConfigurationService.getInt(CONF_USER_RETRY_DEFAULT, 0);}}
line. I think we only need to hardwire it only into the xml. Please check
OOZIE-1890 and OOZIE-3462 for the reasons.
* Can you please add some meaninful assert messages to your unit test asserts.
> Possibility to set retry-max globally
> -
>
> Key: OOZIE-3066
> URL: https://issues.apache.org/jira/browse/OOZIE-3066
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: trunk
>Reporter: Gézapeti
>Assignee: Qs Bao
>Priority: Major
> Attachments: OOZIE-3066-001.patch
>
>
> Currently there is no way to set the equivalent of {{retry-max}} globally to
> apply to every action.
> It should be possible to set a default value to that to make workflow
> execution more reliable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3569) SSH Action should add checking success file
[
https://issues.apache.org/jira/browse/OOZIE-3569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010484#comment-17010484
]
Andras Salamon commented on OOZIE-3569:
---
Thanks for the patch [~zuston] I like this approach.
The success file checking is rather similar to the error file checking (which
is good of course), could you please extract this to a new method?
Have you checked your modification in a working environment? We don't have a
working ssh action unit test, so it would be important to check it.
> SSH Action should add checking success file
> ---
>
> Key: OOZIE-3569
> URL: https://issues.apache.org/jira/browse/OOZIE-3569
> Project: Oozie
> Issue Type: Bug
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Attachments: OOZIE-3569-v1.patch
>
>
> *Phenomenon*
> Currently, {{SSH Action}} checking operation are as following:
> Firstly, check operation is to check {{Oozie}} ppid. When pgid does not
> exist, check whether there is an error file. If the error file does not
> exist, {{Oozie}} will set action status {{OK}}
> However, when {{Oozie}} pgid is killed externally, this action will be
> incorrectly determined to be successful.
> *Solution*
> In ssh-wrapper.sh, when command execution is OK, {{Oozie}} should touch a
> success empty file like touching error file.
> In {{SshActionExecutor}} check method, Oozie should add checking the success
> file existence.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
