[jira] [Created] (OOZIE-1623) JPAService doesn't need to do reads in a transaction
Robert Kanter created OOZIE-1623:
Summary: JPAService doesn't need to do reads in a transaction
Key: OOZIE-1623
URL: https://issues.apache.org/jira/browse/OOZIE-1623
Project: Oozie
Issue Type: Bug
Affects Versions: trunk
Reporter: Robert Kanter
Assignee: Robert Kanter
{{JPAService#executeGet}} and {{JPAService#executeGetList}} are both doing
{{SELECT}} queries (so only reading), which means that they shouldn't need to
be in a transaction.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (OOZIE-1574) Create command line option for oozie sqoop CLI
[
https://issues.apache.org/jira/browse/OOZIE-1574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13830346#comment-13830346
]
Robert Kanter commented on OOZIE-1574:
--
Let's move all discussion over to OOZIE-1575 and mark this as a duplicate.
Also, if you put it in \{noformat\} or \{code\} tags, it won't mess it up.
{noformat}
/user/${wf:user()}/${examplesRoot}/output-data/sqoop
{noformat}
Create command line option for oozie sqoop CLI
--
Key: OOZIE-1574
URL: https://issues.apache.org/jira/browse/OOZIE-1574
Project: Oozie
Issue Type: Sub-task
Components: client
Reporter: Bowen Zhang
Assignee: Bowen Zhang
Fix For: trunk
Attachments: oozie-1574.patch
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (OOZIE-1575) Add functionality to sumbit sqoop job through http on oozie server side
[
https://issues.apache.org/jira/browse/OOZIE-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13830345#comment-13830345
]
Robert Kanter commented on OOZIE-1575:
--
1. I don't think we want to keep renaming the {{SubmitScriptLanguageXCommand}}
class every time we add a new command or we'll end up with something like
{{SubmitScriptLanguageOrSqoopOrDistCpOrWhateverOrSomethingElseXCommand}}.
Clearly some of the stuff written in {{SubmitScriptLanguageXCommand}} is useful
for other types of actions, so it should be moved to the parent
{{SubmitHttpXCommand}}, and then everything can inherit them.
2. In that case, there's no reason to store them in a list
From OOZIE-1574:
{quote}
The sqoop -file option points to the file that contains sqoop command to run
where each option occupies its own line.
{quote}
3. When using the Sqoop action normally, we don't let the user put the Sqoop
command in a file; it would be best if the Oozie CLI method of submitting a
Sqoop job were mostly the same as the regular way. Is there some way to have
the Sqoop command be part of the command line arguments instead of as a
separate file?
Add functionality to sumbit sqoop job through http on oozie server side
---
Key: OOZIE-1575
URL: https://issues.apache.org/jira/browse/OOZIE-1575
Project: Oozie
Issue Type: Sub-task
Components: client
Reporter: Bowen Zhang
Assignee: Bowen Zhang
Fix For: trunk
Attachments: oozie-1575.patch, oozie-1575.patch, oozie-1575.patch
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (OOZIE-1616) Add sharelib and launcherlib locations to the instrumentation info
[ https://issues.apache.org/jira/browse/OOZIE-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13830377#comment-13830377 ] purshotam shah commented on OOZIE-1616: --- Comment on patch: This may not work for sharelib property file configuration, we need to get the content of property file and print them as sharelib path. Add sharelib and launcherlib locations to the instrumentation info -- Key: OOZIE-1616 URL: https://issues.apache.org/jira/browse/OOZIE-1616 Project: Oozie Issue Type: Sub-task Affects Versions: trunk Reporter: Robert Kanter Assignee: Robert Kanter Attachments: OOZIE-1616.patch It would be convenient to add the sharelib and launcher lib locations to the instrumentation info reported by Oozie. This way, users can easily see which sharelib they are currently using. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (OOZIE-1624) Exclusion pattern for sharelib.
purshotam shah created OOZIE-1624: - Summary: Exclusion pattern for sharelib. Key: OOZIE-1624 URL: https://issues.apache.org/jira/browse/OOZIE-1624 Project: Oozie Issue Type: Bug Reporter: purshotam shah -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (OOZIE-1624) Exclusion pattern for sharelib.
[ https://issues.apache.org/jira/browse/OOZIE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] purshotam shah updated OOZIE-1624: -- Issue Type: Sub-task (was: Bug) Parent: OOZIE-1619 Exclusion pattern for sharelib. --- Key: OOZIE-1624 URL: https://issues.apache.org/jira/browse/OOZIE-1624 Project: Oozie Issue Type: Sub-task Reporter: purshotam shah Sharelib may bring some jar which might conflict with user jars. Ex. Sharelib hive has json-2..jar, where as some of the user use-case need higher version of json jar. He should be able to exclude sharelib json jar and bring his own version. property nameoozie.action.sharelib.exclusion/name valuejson-*.jar,abc-*.jar/value /property -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (OOZIE-1612) When printing Dates to log messages, we should make sure they are in oozie.processing.timezone
[
https://issues.apache.org/jira/browse/OOZIE-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated OOZIE-1612:
Attachment: oozie-1612.2.patch
I didn't find an efficient way to format Date objects within XLog.
In addition, I noticed that my previous patch would only correct timezone if
formatting template is used for logging. In 100% of the places where Dates are
logged, templates are not used - instead the Date objects are concatenated into
a string and then logged. Which means the formatter will never be called and
the wrong timezone will be used.
So, attaching the ugly patch: multiple locations where Dates are logged are
converted to Oozie timezone. And one location with a comment explaining why I
chose not to convert :)
When printing Dates to log messages, we should make sure they are in
oozie.processing.timezone
--
Key: OOZIE-1612
URL: https://issues.apache.org/jira/browse/OOZIE-1612
Project: Oozie
Issue Type: Improvement
Affects Versions: trunk
Reporter: Robert Kanter
Priority: Minor
Labels: newbie
Attachments: OOZIE-1612.1.patch, OOZIE-1612.patch, oozie-1612.2.patch
We were recently looking into an issue and noticed that the same log message
had printed different date objects with different timezones, which makes it
hard to compare the two. Which leads to the bigger picture, which is that we
should be printing any Date objects in log messages with the
{{oozie.processing.timezone}} timezone (there's a method for that in
{{DateUtils}}).
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (OOZIE-1575) Add functionality to sumbit sqoop job through http on oozie server side
[ https://issues.apache.org/jira/browse/OOZIE-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13830428#comment-13830428 ] Bowen Zhang commented on OOZIE-1575: If we want to just pass in the actual sqoop command, then, I need to write a custom sqoop parser for handling both regular and free-form. Add functionality to sumbit sqoop job through http on oozie server side --- Key: OOZIE-1575 URL: https://issues.apache.org/jira/browse/OOZIE-1575 Project: Oozie Issue Type: Sub-task Components: client Reporter: Bowen Zhang Assignee: Bowen Zhang Fix For: trunk Attachments: oozie-1575.patch, oozie-1575.patch, oozie-1575.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: Review Request 14239: OOZIE-1504 parameterize coord EL functions.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14239/
---
(Updated Nov. 23, 2013, 12:24 a.m.)
Review request for oozie.
Changes
---
Review comments.
Bugs: OOZIE-1504
https://issues.apache.org/jira/browse/OOZIE-1504
Repository: oozie
Description (updated)
---
Use-case
-
A workflow that processes input datasets for the past 30 days:
input-events
data-in name=event_input_path_format1 dataset=EVENT_INPUT_FORMAT1
start-instance$
{coord:current(-30)}
/start-instance
end-instance$
{coord:current(-1)}
/end-instance
/data-in
data-in name=event_input_path_format2 dataset=EVENT_INPUT_FORMAT2
start-instance$
{coord:current(-30)}
/start-instance
end-instance$
{coord:current(-1)}
/end-instance
/data-in
/input-events
Instead, one wants it to start on the same specific date so that each day, it
processes one more data set than the last day (the datasets are daily).
Something like the following is not supported
input-events
data-in name=event_input_path_format1 dataset=EVENT_INPUT_FORMAT1
start-instance${coord:absolute(2013-03-15T00:00Z)}/start-instance
end-instance$
{coord:current(-1)}
/end-instance
/data-in
data-in name=event_input_path_format2 dataset=EVENT_INPUT_FORMAT2
start-instance${coord:absolute(2013-03-15T00:00Z)}/start-instance
end-instance$
{coord:current(-1)}
/end-instance
/data-in
/input-events
-
Fix support providing absolute value for instance range. One can have a
combination of absolute date with coord:current().
Other EL function are not supported with absolute date.
coord:Current() can be parameterize.
like
end-instance${coord:latest(end_date)}/end-instance // where end_date is
one of the specified property.
Diffs (updated)
-
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java
1544636
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java
1544636
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml
1544636
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordCommandUtils.java
1544636
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java
1544636
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-1.xml
PRE-CREATION
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-2.xml
PRE-CREATION
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-3.xml
PRE-CREATION
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization.xml
PRE-CREATION
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/CoordinatorFunctionalSpec.twiki
1544636
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/WebServicesAPI.twiki
1544636
Diff: https://reviews.apache.org/r/14239/diff/
Testing
---
Added multiple test cases to verify combination of scenarios.
Thanks,
Purshotam Shah
[jira] [Commented] (OOZIE-1575) Add functionality to sumbit sqoop job through http on oozie server side
[ https://issues.apache.org/jira/browse/OOZIE-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13830473#comment-13830473 ] Bowen Zhang commented on OOZIE-1575: Are you proposing passing a command line within the command line? like: oozie sqoop -command import connect jdbc --query 'select * from T' -m -1 -config config.properties -Dmapred.queue.name=default. Add functionality to sumbit sqoop job through http on oozie server side --- Key: OOZIE-1575 URL: https://issues.apache.org/jira/browse/OOZIE-1575 Project: Oozie Issue Type: Sub-task Components: client Reporter: Bowen Zhang Assignee: Bowen Zhang Fix For: trunk Attachments: oozie-1575.patch, oozie-1575.patch, oozie-1575.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: Review Request 15225: OOZIE-1581 Workflow performance optimizations
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15225/ --- (Updated Nov. 23, 2013, 2:36 a.m.) Review request for oozie. Changes --- Addressed review comments. Note - synchronous calling will cause NPE in CoordActionUpdate command, if the workflow is a no-op wf. no-op workflows are not real use-case anyway, so the examples which had no-op wf have also been updated to have an empty single action (almost no-op). The patch also renames the dir 'examples/src/main/apps/no-op' to 'examples/src/main/apps/one-op', but svn is giving some pain to include a rename in the patch. Will remove and add dir while committing. Bugs: OOZIE-1581 https://issues.apache.org/jira/browse/OOZIE-1581 Repository: oozie Description --- Patch for review updated from the attachments on JIRA. 1. Included change to call coordinator action Start synchronously from Ready command. 2. Made Decision Action Executor extend from Control Action Executor. That way the synchronous commands can be executed for all such 'connector' actions. 3. Fixed failing unit tests. 4. TestSLAEventGeneration unit test is failing - WIP. Diffs (updated) - trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1544635 trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1544635 trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1544635 trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1544635 trunk/core/src/main/java/org/apache/oozie/command/wf/ActionCheckXCommand.java 1544635 trunk/core/src/main/java/org/apache/oozie/command/wf/ActionEndXCommand.java 1544635 trunk/core/src/main/java/org/apache/oozie/command/wf/ActionStartXCommand.java 1544635 trunk/core/src/main/java/org/apache/oozie/command/wf/SignalXCommand.java 1544635 trunk/core/src/main/java/org/apache/oozie/service/WorkflowStoreService.java 1544635 trunk/core/src/test/java/org/apache/oozie/action/TestActionFailover.java 1544635 trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1544635 trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionNotificationXCommand.java 1544635 trunk/core/src/test/java/org/apache/oozie/command/wf/TestNotificationXCommand.java 1544635 trunk/core/src/test/java/org/apache/oozie/command/wf/TestSignalXCommand.java 1544635 trunk/core/src/test/java/org/apache/oozie/event/TestEventGeneration.java 1544635 trunk/core/src/test/java/org/apache/oozie/service/TestRecoveryService.java 1544635 trunk/core/src/test/java/org/apache/oozie/sla/TestSLAEventGeneration.java 1544635 trunk/core/src/test/java/org/apache/oozie/test/XDataTestCase.java 1544635 trunk/core/src/test/resources/wf-fork.xml 1544635 trunk/core/src/test/resources/wf-no-op.xml 1544635 trunk/examples/src/main/apps/cron-schedule/workflow.xml 1544635 trunk/examples/src/main/apps/cron/workflow.xml 1544635 trunk/examples/src/main/apps/sla/workflow.xml 1544635 Diff: https://reviews.apache.org/r/15225/diff/ Testing --- unit tests fixed. stress testing performed to validate the fast start case. Thanks, Mona Chitnis
[jira] [Commented] (OOZIE-1612) When printing Dates to log messages, we should make sure they are in oozie.processing.timezone
[
https://issues.apache.org/jira/browse/OOZIE-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13830539#comment-13830539
]
Hadoop QA commented on OOZIE-1612:
--
Testing JIRA OOZIE-1612
Cleaning local svn workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 3 line(s) longer than 132
characters
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1368
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/903/
When printing Dates to log messages, we should make sure they are in
oozie.processing.timezone
--
Key: OOZIE-1612
URL: https://issues.apache.org/jira/browse/OOZIE-1612
Project: Oozie
Issue Type: Improvement
Affects Versions: trunk
Reporter: Robert Kanter
Priority: Minor
Labels: newbie
Attachments: OOZIE-1612.1.patch, OOZIE-1612.patch, oozie-1612.2.patch
We were recently looking into an issue and noticed that the same log message
had printed different date objects with different timezones, which makes it
hard to compare the two. Which leads to the bigger picture, which is that we
should be printing any Date objects in log messages with the
{{oozie.processing.timezone}} timezone (there's a method for that in
{{DateUtils}}).
--
This message was sent by Atlassian JIRA
(v6.1#6144)
