Re: Review Request: OOZIE-1186 Image load for Job DAG visualization should handle resources better
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9079/#review15647 --- Looks good. Just need one minor change to deal with java bug. trunk/core/src/main/java/org/apache/oozie/util/GraphGenerator.java https://reviews.apache.org/r/9079/#comment33747 Can we also add a out = null after close. http://wiki.apache.org/tomcat/FAQ/KnownIssues#ImageIOIssues - Rohini Palaniswamy On Jan. 24, 2013, 1:59 a.m., Mona Chitnis wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9079/ --- (Updated Jan. 24, 2013, 1:59 a.m.) Review request for oozie. Description --- https://issues.apache.org/jira/browse/OOZIE-1186 This addresses bug OOZIE-1186. https://issues.apache.org/jira/browse/OOZIE-1186 Diffs - trunk/core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java 1437616 trunk/core/src/main/java/org/apache/oozie/util/GraphGenerator.java 1437616 trunk/core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java 1437616 Diff: https://reviews.apache.org/r/9079/diff/ Testing --- unit test done + end-to-end using Yourkit memory profiler Thanks, Mona Chitnis
Re: Review Request: OOZIE-1186 Image load for Job DAG visualization should handle resources better
On Jan. 24, 2013, 5:33 p.m., Rohini Palaniswamy wrote: trunk/core/src/main/java/org/apache/oozie/util/GraphGenerator.java, line 263 https://reviews.apache.org/r/9079/diff/1/?file=251437#file251437line263 Can we also add a out = null after close. http://wiki.apache.org/tomcat/FAQ/KnownIssues#ImageIOIssues Actually you need to do the out.close() and out = null in finally also. - Rohini --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9079/#review15647 --- On Jan. 24, 2013, 1:59 a.m., Mona Chitnis wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9079/ --- (Updated Jan. 24, 2013, 1:59 a.m.) Review request for oozie. Description --- https://issues.apache.org/jira/browse/OOZIE-1186 This addresses bug OOZIE-1186. https://issues.apache.org/jira/browse/OOZIE-1186 Diffs - trunk/core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java 1437616 trunk/core/src/main/java/org/apache/oozie/util/GraphGenerator.java 1437616 trunk/core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java 1437616 Diff: https://reviews.apache.org/r/9079/diff/ Testing --- unit test done + end-to-end using Yourkit memory profiler Thanks, Mona Chitnis
[jira] [Created] (OOZIE-1188) Typo in documentation for using login server example
Robert Kanter created OOZIE-1188:
Summary: Typo in documentation for using login server example
Key: OOZIE-1188
URL: https://issues.apache.org/jira/browse/OOZIE-1188
Project: Oozie
Issue Type: Bug
Components: docs
Affects Versions: trunk
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Trivial
Fix For: trunk
The docs say to use the property {{alt-kerberos.non-browser.user-agents}} to
configure which user-agents are considered not to be browsers; this won't work
because the correct property is
{{oozie.authentication.alt-kerberos.non-browser.user-agents}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OOZIE-1188) Typo in documentation for using login server example
[
https://issues.apache.org/jira/browse/OOZIE-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated OOZIE-1188:
-
Attachment: OOZIE-1188.patch
Typo in documentation for using login server example
Key: OOZIE-1188
URL: https://issues.apache.org/jira/browse/OOZIE-1188
Project: Oozie
Issue Type: Bug
Components: docs
Affects Versions: trunk
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Trivial
Fix For: trunk
Attachments: OOZIE-1188.patch
The docs say to use the property {{alt-kerberos.non-browser.user-agents}} to
configure which user-agents are considered not to be browsers; this won't
work because the correct property is
{{oozie.authentication.alt-kerberos.non-browser.user-agents}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-1188) Typo in documentation for using login server example
[
https://issues.apache.org/jira/browse/OOZIE-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13561887#comment-13561887
]
Alejandro Abdelnur commented on OOZIE-1188:
---
+1 (no need to wait for testpatch, this is just a doc fix)
Typo in documentation for using login server example
Key: OOZIE-1188
URL: https://issues.apache.org/jira/browse/OOZIE-1188
Project: Oozie
Issue Type: Bug
Components: docs
Affects Versions: trunk
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Trivial
Fix For: trunk
Attachments: OOZIE-1188.patch
The docs say to use the property {{alt-kerberos.non-browser.user-agents}} to
configure which user-agents are considered not to be browsers; this won't
work because the correct property is
{{oozie.authentication.alt-kerberos.non-browser.user-agents}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-1186) Image load for Job DAG visualization should handle resources better
[ https://issues.apache.org/jira/browse/OOZIE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13561931#comment-13561931 ] Virag Kothari commented on OOZIE-1186: -- I did some testing to figure out the effect of flush() I created a bufferedimage with width and height of 3000*3000. As RGB is used, each pixel occupies a 'int'. So the buffered image instance occupies around (3000*3000*4/(1024*1024) MB. I then tried with and without img.flush() but no difference in memory consumption to make any conclusion. Basically, only the garbage collector will reclaim the databuffer when the img instance is out of scope. So, ideally we should be creating a image which can be written serially to disk (row by row) instead of bringing it entirely in memory. But that would be non-trivial with the graph library being used. So, short term fix would be increase the tomcat memory if OOM's are seen. But, it would be good to have flush() which will flush any system resources used for rendering the image. dispose() is not required as its already called by the component's paintAll() method. Image load for Job DAG visualization should handle resources better --- Key: OOZIE-1186 URL: https://issues.apache.org/jira/browse/OOZIE-1186 Project: Oozie Issue Type: Bug Affects Versions: trunk, 3.3.1 Reporter: Mona Chitnis Assignee: Mona Chitnis Fix For: trunk, 3.3.1 Attachments: HeapMemoryExistingTrunkCode.png, HeapMemoryWithPatchedCode.png, OOZIE-1186.patch The Job DAG visualization loads an image into memory to be streamed on outputstream. However, it does not free up memory and I/O resources leading to 'Out of Java heap space' errors. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-1188) Typo in documentation for using login server example
[
https://issues.apache.org/jira/browse/OOZIE-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13561940#comment-13561940
]
Hadoop QA commented on OOZIE-1188:
--
Testing JIRA OOZIE-1188
Cleaning local svn workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 947
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/295/
Typo in documentation for using login server example
Key: OOZIE-1188
URL: https://issues.apache.org/jira/browse/OOZIE-1188
Project: Oozie
Issue Type: Bug
Components: docs
Affects Versions: trunk
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Trivial
Fix For: trunk
Attachments: OOZIE-1188.patch
The docs say to use the property {{alt-kerberos.non-browser.user-agents}} to
configure which user-agents are considered not to be browsers; this won't
work because the correct property is
{{oozie.authentication.alt-kerberos.non-browser.user-agents}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: OOZIE-1186 Image load for Job DAG visualization should handle resources better
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9079/#review15651 --- trunk/core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java https://reviews.apache.org/r/9079/#comment33767 Test case is not required. We can remove the extra boolean parameter if it was just for testcase. trunk/core/src/main/java/org/apache/oozie/util/GraphGenerator.java https://reviews.apache.org/r/9079/#comment33755 not required trunk/core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java https://reviews.apache.org/r/9079/#comment33763 The test case fails/passes inconsistently. I also tried with thread.sleep() to allow some time for the memory to be freed but doesn't work. System.gc() is just an indicator for garbage collection and wont help Its ok to not have a testcase as its difficult to write a unit test for a patch which is just flushing system resources and not solving a memory leak issue. - Virag Kothari On Jan. 24, 2013, 1:59 a.m., Mona Chitnis wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9079/ --- (Updated Jan. 24, 2013, 1:59 a.m.) Review request for oozie. Description --- https://issues.apache.org/jira/browse/OOZIE-1186 This addresses bug OOZIE-1186. https://issues.apache.org/jira/browse/OOZIE-1186 Diffs - trunk/core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java 1437616 trunk/core/src/main/java/org/apache/oozie/util/GraphGenerator.java 1437616 trunk/core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java 1437616 Diff: https://reviews.apache.org/r/9079/diff/ Testing --- unit test done + end-to-end using Yourkit memory profiler Thanks, Mona Chitnis
[jira] [Commented] (OOZIE-1178) Workflow Application Master in YARN
[ https://issues.apache.org/jira/browse/OOZIE-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13562018#comment-13562018 ] Andrew Purtell commented on OOZIE-1178: --- Hi [~acmurthy]. I'd love to but I must decline, unfortunately I will have full time commitments to HBase and HDFS issues at least through 2013. Workflow Application Master in YARN --- Key: OOZIE-1178 URL: https://issues.apache.org/jira/browse/OOZIE-1178 Project: Oozie Issue Type: New Feature Reporter: Bo Wang Attachments: MAPREDUCE-4495-v1.1.patch, MAPREDUCE-4495-v1.patch, MapReduceWorkflowAM.pdf, yapp_proposal.txt It is useful to have a workflow application master, which will be capable of running a DAG of jobs. The workflow client submits a DAG request to the AM and then the AM will manage the life cycle of this application in terms of requesting the needed resources from the RM, and starting, monitoring and retrying the application's individual tasks. Compared to running Oozie with the current MapReduce Application Master, these are some of the advantages: - Less number of consumed resources, since only one application master will be spawned for the whole workflow. - Reuse of resources, since the same resources can be used by multiple consecutive jobs in the workflow (no need to request/wait for resources for every individual job from the central RM). - More optimization opportunities in terms of collective resource requests. - Optimization opportunities in terms of rewriting and composing jobs in the workflow (e.g. pushing down Mappers). - This Application Master can be reused/extended by higher systems like Pig and hive to provide an optimized way of running their workflows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: OOZIE-1186 Image load for Job DAG visualization should handle resources better
On Jan. 24, 2013, 8:22 p.m., Virag Kothari wrote: trunk/core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java, line 113 https://reviews.apache.org/r/9079/diff/1/?file=251438#file251438line113 The test case fails/passes inconsistently. I also tried with thread.sleep() to allow some time for the memory to be freed but doesn't work. System.gc() is just an indicator for garbage collection and wont help Its ok to not have a testcase as its difficult to write a unit test for a patch which is just flushing system resources and not solving a memory leak issue. commenting the testcase. Letting code be there for reference - Mona --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9079/#review15651 --- On Jan. 24, 2013, 1:59 a.m., Mona Chitnis wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9079/ --- (Updated Jan. 24, 2013, 1:59 a.m.) Review request for oozie. Description --- https://issues.apache.org/jira/browse/OOZIE-1186 This addresses bug OOZIE-1186. https://issues.apache.org/jira/browse/OOZIE-1186 Diffs - trunk/core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java 1437616 trunk/core/src/main/java/org/apache/oozie/util/GraphGenerator.java 1437616 trunk/core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java 1437616 Diff: https://reviews.apache.org/r/9079/diff/ Testing --- unit test done + end-to-end using Yourkit memory profiler Thanks, Mona Chitnis
Re: Review Request: OOZIE-1186 Image load for Job DAG visualization should handle resources better
On Jan. 24, 2013, 8:22 p.m., Virag Kothari wrote: trunk/core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java, line 113 https://reviews.apache.org/r/9079/diff/1/?file=251438#file251438line113 The test case fails/passes inconsistently. I also tried with thread.sleep() to allow some time for the memory to be freed but doesn't work. System.gc() is just an indicator for garbage collection and wont help Its ok to not have a testcase as its difficult to write a unit test for a patch which is just flushing system resources and not solving a memory leak issue. Mona Chitnis wrote: commenting the testcase. Letting code be there for reference Mohammad had once told me not to have commented code. You can put a TODO comment or a note for reference - Virag --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9079/#review15651 --- On Jan. 24, 2013, 1:59 a.m., Mona Chitnis wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9079/ --- (Updated Jan. 24, 2013, 1:59 a.m.) Review request for oozie. Description --- https://issues.apache.org/jira/browse/OOZIE-1186 This addresses bug OOZIE-1186. https://issues.apache.org/jira/browse/OOZIE-1186 Diffs - trunk/core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java 1437616 trunk/core/src/main/java/org/apache/oozie/util/GraphGenerator.java 1437616 trunk/core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java 1437616 Diff: https://reviews.apache.org/r/9079/diff/ Testing --- unit test done + end-to-end using Yourkit memory profiler Thanks, Mona Chitnis
[jira] [Updated] (OOZIE-1181) Dependency cache with configurations for eviction, ttl and max elements in memory
[ https://issues.apache.org/jira/browse/OOZIE-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohini Palaniswamy updated OOZIE-1181: -- Attachment: OOZIE-1181-2.patch Dependency cache with configurations for eviction, ttl and max elements in memory - Key: OOZIE-1181 URL: https://issues.apache.org/jira/browse/OOZIE-1181 Project: Oozie Issue Type: Sub-task Reporter: Rohini Palaniswamy Assignee: Rohini Palaniswamy Fix For: trunk Attachments: OOZIE-1181-1.patch, OOZIE-1181-2.patch SimpleHCatDependencyCache is just a in memory map implementation. Writing an alternate implemenation using ehcache which allows configurations for eviction based on ttl or time to idle and max elements in memory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: [OOZIE-1181] Dependency cache with configurations for eviction, ttl and max elements in memory
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9026/ --- (Updated Jan. 24, 2013, 9:51 p.m.) Review request for oozie. Changes --- Fixed/removed tests that had transient failures. Description --- Cache based on ehcache which has eviction, ttl, max elements in memory and overflow to disk configurable Diffs (updated) - http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/pom.xml 1437215 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/main/java/org/apache/oozie/dependency/cache/EhcacheHCatDependencyCache.java PRE-CREATION http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/main/java/org/apache/oozie/dependency/cache/WaitingActions.java PRE-CREATION http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/main/resources/ehcache-default.xml PRE-CREATION http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/test/java/org/apache/oozie/service/TestPartitionDependencyManagerEhcache.java PRE-CREATION http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/test/java/org/apache/oozie/service/TestPartitionDependencyManagerService.java 1437215 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/test/resources/ehcache.xml PRE-CREATION http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/pom.xml 1437215 Diff: https://reviews.apache.org/r/9026/diff/ Testing --- Thanks, Rohini Palaniswamy
Re: Review Request: [OOZIE-1185] Retry jms connections on failure
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9061/ --- (Updated Jan. 24, 2013, 9:58 p.m.) Review request for oozie. Changes --- Fixed mkdistro failure. Description --- Retry jms connections on failure backing off exponentially (1min,2 min, 4,8,16..) up to a max limit This addresses bug OOZIE-1185. https://issues.apache.org/jira/browse/OOZIE-1185 Diffs (updated) - http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/main/java/org/apache/oozie/dependency/FSURIHandler.java 1438212 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/main/java/org/apache/oozie/dependency/HCatURIHandler.java 1438212 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/main/java/org/apache/oozie/jms/DefaultConnectionContext.java 1438212 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/main/java/org/apache/oozie/jms/JMSExceptionListener.java PRE-CREATION http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/main/java/org/apache/oozie/service/JMSAccessorService.java 1438212 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/main/java/org/apache/oozie/service/SchedulerService.java 1438212 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/test/java/org/apache/oozie/service/TestJMSAccessorService.java 1438212 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/test/java/org/apache/oozie/test/MiniHCatServer.java 1438212 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/hcataloglibs/hcatalog-0.6.0-SNAPSHOT/pom.xml 1438212 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/hcataloglibs/hcatalog-0.6/pom.xml PRE-CREATION http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/hcataloglibs/pom.xml 1438212 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/pom.xml 1438212 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/src/main/assemblies/hcataloglib.xml 1438212 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/src/main/assemblies/hcataloglibs.xml 1438212 Diff: https://reviews.apache.org/r/9061/diff/ Testing --- Thanks, Rohini Palaniswamy
[jira] [Updated] (OOZIE-959) Use API from OOZIE-906 in console
[ https://issues.apache.org/jira/browse/OOZIE-959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohini Palaniswamy updated OOZIE-959: - Fix Version/s: (was: trunk) 3.3.0 Use API from OOZIE-906 in console - Key: OOZIE-959 URL: https://issues.apache.org/jira/browse/OOZIE-959 Project: Oozie Issue Type: Improvement Affects Versions: 3.3.0, trunk Reporter: Ashish Vaidya Assignee: Ashish Vaidya Labels: webconsole Fix For: 3.3.0 Attachments: OOZIE-959.branch-3.3.patch, OOZIE-959.trunk.patch Update Oozie web console (written in ExtJS) to consume the new runtime DAG plotting API in JIRA OOZIE-906. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: [OOZIE-1185] Retry jms connections on failure
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9061/#review15659 --- Ship it! +1..looks good...couple of comments below http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/main/java/org/apache/oozie/jms/JMSExceptionListener.java https://reviews.apache.org/r/9061/#comment33783 javadoc here and for other public stuff http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/main/java/org/apache/oozie/service/JMSAccessorService.java https://reviews.apache.org/r/9061/#comment33781 this method may be called simultaneously for the same jmsconnectString so we should only schedule retry for a connection if its not already done http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/main/java/org/apache/oozie/service/SchedulerService.java https://reviews.apache.org/r/9061/#comment33782 Can we reuse the existing methods of Scheduler SErvice - Virag Kothari On Jan. 24, 2013, 9:58 p.m., Rohini Palaniswamy wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9061/ --- (Updated Jan. 24, 2013, 9:58 p.m.) Review request for oozie. Description --- Retry jms connections on failure backing off exponentially (1min,2 min, 4,8,16..) up to a max limit This addresses bug OOZIE-1185. https://issues.apache.org/jira/browse/OOZIE-1185 Diffs - http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/main/java/org/apache/oozie/dependency/FSURIHandler.java 1438212 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/main/java/org/apache/oozie/dependency/HCatURIHandler.java 1438212 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/main/java/org/apache/oozie/jms/DefaultConnectionContext.java 1438212 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/main/java/org/apache/oozie/jms/JMSExceptionListener.java PRE-CREATION http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/main/java/org/apache/oozie/service/JMSAccessorService.java 1438212 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/main/java/org/apache/oozie/service/SchedulerService.java 1438212 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/test/java/org/apache/oozie/service/TestJMSAccessorService.java 1438212 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/test/java/org/apache/oozie/test/MiniHCatServer.java 1438212 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/hcataloglibs/hcatalog-0.6.0-SNAPSHOT/pom.xml 1438212 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/hcataloglibs/hcatalog-0.6/pom.xml PRE-CREATION http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/hcataloglibs/pom.xml 1438212 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/pom.xml 1438212 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/src/main/assemblies/hcataloglib.xml 1438212 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/src/main/assemblies/hcataloglibs.xml 1438212 Diff: https://reviews.apache.org/r/9061/diff/ Testing --- Thanks, Rohini Palaniswamy
[jira] [Commented] (OOZIE-1187) reduce memory usage of SLA query (invoked by CLI command) to avoid OOM
[ https://issues.apache.org/jira/browse/OOZIE-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13562144#comment-13562144 ] Ryota Egashira commented on OOZIE-1187: --- patch uploaded https://reviews.apache.org/r/9104/ reduce memory usage of SLA query (invoked by CLI command) to avoid OOM -- Key: OOZIE-1187 URL: https://issues.apache.org/jira/browse/OOZIE-1187 Project: Oozie Issue Type: Bug Components: core Affects Versions: trunk Reporter: Ryota Egashira Assignee: Ryota Egashira Labels: oozie Fix For: trunk oozie sla -len 1000 caused OOM in Y! setting. this jira to do following 1) use JDBCFetchPlan to reduce memory usage in SLAEventsGetJPAExecutor -just like we are doing for oozie jobs command 2) enforce max cap(say, 1000) in -len parameter that user sets -need documentation change to notify user -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: [OOZIE-1185] Retry jms connections on failure
On Jan. 24, 2013, 11:20 p.m., Virag Kothari wrote: http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/main/java/org/apache/oozie/jms/JMSExceptionListener.java, line 27 https://reviews.apache.org/r/9061/diff/2/?file=251720#file251720line27 javadoc here and for other public stuff Done. Did not do for methods with @Override as it is not required. On Jan. 24, 2013, 11:20 p.m., Virag Kothari wrote: http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/main/java/org/apache/oozie/service/SchedulerService.java, line 188 https://reviews.apache.org/r/9061/diff/2/?file=251722#file251722line188 Can we reuse the existing methods of Scheduler SErvice No. Want to schedule only once after a delay. Existing methods do it repeatedly based on a interval - Rohini --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9061/#review15659 --- On Jan. 25, 2013, 12:03 a.m., Rohini Palaniswamy wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9061/ --- (Updated Jan. 25, 2013, 12:03 a.m.) Review request for oozie. Description --- Retry jms connections on failure backing off exponentially (1min,2 min, 4,8,16..) up to a max limit This addresses bug OOZIE-1185. https://issues.apache.org/jira/browse/OOZIE-1185 Diffs - http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/main/java/org/apache/oozie/dependency/FSURIHandler.java 1438270 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/main/java/org/apache/oozie/dependency/HCatURIHandler.java 1438270 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/main/java/org/apache/oozie/jms/DefaultConnectionContext.java 1438270 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/main/java/org/apache/oozie/jms/JMSExceptionListener.java PRE-CREATION http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/main/java/org/apache/oozie/service/JMSAccessorService.java 1438270 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/main/java/org/apache/oozie/service/SchedulerService.java 1438270 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/test/java/org/apache/oozie/service/TestJMSAccessorService.java 1438270 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/core/src/test/java/org/apache/oozie/test/MiniHCatServer.java 1438270 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/hcataloglibs/hcatalog-0.6.0-SNAPSHOT/pom.xml 1438270 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/hcataloglibs/hcatalog-0.6/pom.xml PRE-CREATION http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/hcataloglibs/pom.xml 1438270 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/pom.xml 1438270 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/src/main/assemblies/hcataloglib.xml 1438270 http://svn.apache.org/repos/asf/oozie/branches/hcat-intre/src/main/assemblies/hcataloglibs.xml 1438270 Diff: https://reviews.apache.org/r/9061/diff/ Testing --- Thanks, Rohini Palaniswamy
[jira] [Updated] (OOZIE-1185) Retry jms connections on failure
[ https://issues.apache.org/jira/browse/OOZIE-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohini Palaniswamy updated OOZIE-1185: -- Attachment: OOZIE-1185-3.patch OOZIE-1185-2.patch Retry jms connections on failure Key: OOZIE-1185 URL: https://issues.apache.org/jira/browse/OOZIE-1185 Project: Oozie Issue Type: Sub-task Reporter: Rohini Palaniswamy Assignee: Rohini Palaniswamy Fix For: trunk Attachments: OOZIE-1185-1.patch, OOZIE-1185-2.patch, OOZIE-1185-3.patch When there is a JMS connection failure, need to reconnect and reregister listening topics. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: reduce memory usage of SLA query (invoked by CLI command) to avoid OOM
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9104/ --- (Updated Jan. 25, 2013, 12:18 a.m.) Review request for oozie. Changes --- added minor doc change Description --- https://issues.apache.org/jira/browse/OOZIE-1187 This addresses bug OOZIE-1187. https://issues.apache.org/jira/browse/OOZIE-1187 Diffs (updated) - trunk/client/src/main/java/org/apache/oozie/cli/OozieCLI.java 1437603 trunk/core/src/main/java/org/apache/oozie/SLAEventBean.java 1437603 trunk/core/src/main/java/org/apache/oozie/command/coord/SLAEventsXCommand.java 1437603 trunk/core/src/main/java/org/apache/oozie/executor/jpa/SLAEventsGetForSeqIdJPAExecutor.java 1437603 trunk/docs/src/site/twiki/DG_CommandLineTool.twiki 1437603 Diff: https://reviews.apache.org/r/9104/diff/ Testing --- locally tested. ran coordinator jobs with SLA tag inserted Thanks, Ryota Egashira
[jira] [Resolved] (OOZIE-1185) Retry jms connections on failure
[ https://issues.apache.org/jira/browse/OOZIE-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virag Kothari resolved OOZIE-1185. -- Resolution: Fixed Committed to hcat-intre. Thanks Rohini Retry jms connections on failure Key: OOZIE-1185 URL: https://issues.apache.org/jira/browse/OOZIE-1185 Project: Oozie Issue Type: Sub-task Reporter: Rohini Palaniswamy Assignee: Rohini Palaniswamy Fix For: trunk Attachments: OOZIE-1185-1.patch, OOZIE-1185-2.patch, OOZIE-1185-3.patch When there is a JMS connection failure, need to reconnect and reregister listening topics. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: reduce memory usage of SLA query (invoked by CLI command) to avoid OOM
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9104/#review15670 --- trunk/core/src/main/java/org/apache/oozie/SLAEventBean.java https://reviews.apache.org/r/9104/#comment33803 It seems that all columns are retrieved which is not different from select *. Can you check whether some of the columns (mainly blob's) are internal to Oozie and not useful for the user? trunk/core/src/main/java/org/apache/oozie/executor/jpa/SLAEventsGetForSeqIdJPAExecutor.java https://reviews.apache.org/r/9104/#comment33804 It will be good to add a test case for this JPA executor - Virag Kothari On Jan. 25, 2013, 12:18 a.m., Ryota Egashira wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9104/ --- (Updated Jan. 25, 2013, 12:18 a.m.) Review request for oozie. Description --- https://issues.apache.org/jira/browse/OOZIE-1187 This addresses bug OOZIE-1187. https://issues.apache.org/jira/browse/OOZIE-1187 Diffs - trunk/client/src/main/java/org/apache/oozie/cli/OozieCLI.java 1437603 trunk/core/src/main/java/org/apache/oozie/SLAEventBean.java 1437603 trunk/core/src/main/java/org/apache/oozie/command/coord/SLAEventsXCommand.java 1437603 trunk/core/src/main/java/org/apache/oozie/executor/jpa/SLAEventsGetForSeqIdJPAExecutor.java 1437603 trunk/docs/src/site/twiki/DG_CommandLineTool.twiki 1437603 Diff: https://reviews.apache.org/r/9104/diff/ Testing --- locally tested. ran coordinator jobs with SLA tag inserted Thanks, Ryota Egashira
Re: Review Request: OOZIE-1186 Image load for Job DAG visualization should handle resources better
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9079/
---
(Updated Jan. 25, 2013, 3:46 a.m.)
Review request for oozie.
Changes
---
To understand the utility of using Graphics.dispose(), I studied some of this
code in detail. Here's some data to back the changes in the patch.
Although, the javadocs indicate that we do not need to invoke g.dispose()
explicitly, looking carefully into the Component class's paint()
implementation, note how it only disposes the new graphics object it creates
(co) using the Graphics context and not the original one (g).
http://javasourcecode.org/html/open-source/jdk/jdk-6u23/javax/swing/JComponent.java.html
(lines 967-1053)
bq
public void paint(Graphics g) {
boolean shouldClearPaintFlags = false;
if ((getWidth() = 0) || (getHeight() = 0)) {
return;
}
Graphics componentGraphics = getComponentGraphics(g);
Graphics co = componentGraphics.create();
finally {
co.dispose();
bq.
Therefore, in our GraphGenerator code, as we are creating a new Graphics
object, we alone are responsible for disposing it. The superimplementation of
paint() in the JUNG library's classes e.g.VisualizationImageServer are not seen
to override paint() in a way to do dispose() either.
The heap memory profiling clearly shows the changes with and w/o the dispose
line. Without it, heap occupied keeps monotonically rising with repeated job
requests, dangerously in the ballpark of GBs, till GC hits. With patch, only
spikes with each request but returns to low immediately and stays in MBs.
Excluded the unit test as it was flaky between different run environments. But
applying formatter introduced few changes to that file. Other review comments
addressed.
Description
---
https://issues.apache.org/jira/browse/OOZIE-1186
This addresses bug OOZIE-1186.
https://issues.apache.org/jira/browse/OOZIE-1186
Diffs (updated)
-
trunk/core/src/main/java/org/apache/oozie/util/GraphGenerator.java 1437616
trunk/core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java
1437616
Diff: https://reviews.apache.org/r/9079/diff/
Testing
---
unit test done + end-to-end using Yourkit memory profiler
Thanks,
Mona Chitnis
[jira] [Updated] (OOZIE-1186) Image load for Job DAG visualization should handle resources better
[ https://issues.apache.org/jira/browse/OOZIE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mona Chitnis updated OOZIE-1186: Attachment: (was: OOZIE-1186.patch) Image load for Job DAG visualization should handle resources better --- Key: OOZIE-1186 URL: https://issues.apache.org/jira/browse/OOZIE-1186 Project: Oozie Issue Type: Bug Affects Versions: trunk, 3.3.1 Reporter: Mona Chitnis Assignee: Mona Chitnis Fix For: trunk, 3.3.1 Attachments: HeapMemoryExistingTrunkCode.png, HeapMemoryWithPatchedCode.png, OOZIE-1186v2.patch The Job DAG visualization loads an image into memory to be streamed on outputstream. However, it does not free up memory and I/O resources leading to 'Out of Java heap space' errors. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (OOZIE-1154) Pig action throwing exception when inserting into Hbase table through pig script
[ https://issues.apache.org/jira/browse/OOZIE-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shobin joseph resolved OOZIE-1154. -- Resolution: Fixed Pig action throwing exception when inserting into Hbase table through pig script Key: OOZIE-1154 URL: https://issues.apache.org/jira/browse/OOZIE-1154 Project: Oozie Issue Type: Bug Components: action, workflow Affects Versions: 3.2.0 Environment: hadoop-2.0.0-mr1-cdh4.1.2,hbase-0.92.1-cdh4.1.2,pig-0.10.0-cdh4.1.2,oozie-3.2.0. Reporter: shobin joseph Labels: hadoop, hbase, oozie, pig Fix For: 3.2.0 Original Estimate: 216h Remaining Estimate: 216h Pig scrit to load data from HDFS to Hbase table was sucessfully executed in psuedo map-reduce mode(using pig -x mapreduce scriptname.pig), but when the same script was executed through oozie workflow, exception is thrown even though data is loaded into Hbase table. So this exception breaks the oozie workflow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-1186) Image load for Job DAG visualization should handle resources better
[
https://issues.apache.org/jira/browse/OOZIE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13562370#comment-13562370
]
Hadoop QA commented on OOZIE-1186:
--
Testing JIRA OOZIE-1186
Cleaning local svn workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 947
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:green}*+1 Overall result, good!, no -1s*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/296/
Image load for Job DAG visualization should handle resources better
---
Key: OOZIE-1186
URL: https://issues.apache.org/jira/browse/OOZIE-1186
Project: Oozie
Issue Type: Bug
Affects Versions: trunk, 3.3.1
Reporter: Mona Chitnis
Assignee: Mona Chitnis
Fix For: trunk, 3.3.1
Attachments: HeapMemoryExistingTrunkCode.png,
HeapMemoryWithPatchedCode.png, OOZIE-1186v2.patch
The Job DAG visualization loads an image into memory to be streamed on
outputstream. However, it does not free up memory and I/O resources leading
to 'Out of Java heap space' errors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: OOZIE-1186 Image load for Job DAG visualization should handle resources better
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9079/#review15679 --- Ship it! +1..Good to know about dispose(). It seems we misread the javadoc. It was probably saying that if we override the paint(Graphics g) of Component, g.dispose() shouldn't be called within the method as the graphics object was not created by us trunk/core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java https://reviews.apache.org/r/9079/#comment33823 remove println statement - Virag Kothari On Jan. 25, 2013, 3:46 a.m., Mona Chitnis wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9079/ --- (Updated Jan. 25, 2013, 3:46 a.m.) Review request for oozie. Description --- https://issues.apache.org/jira/browse/OOZIE-1186 This addresses bug OOZIE-1186. https://issues.apache.org/jira/browse/OOZIE-1186 Diffs - trunk/core/src/main/java/org/apache/oozie/util/GraphGenerator.java 1437616 trunk/core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java 1437616 Diff: https://reviews.apache.org/r/9079/diff/ Testing --- unit test done + end-to-end using Yourkit memory profiler Thanks, Mona Chitnis
