-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20006/
-----------------------------------------------------------
(Updated April 14, 2014, 9:26 p.m.)
Review request for oozie, Ryota Egashira and Rohini Palaniswamy.
Changes
-------
Review comment.
Bugs: OOZIE-1769
https://issues.apache.org/jira/browse/OOZIE-1769
Repository: oozie-git
Description
-------
To change the coord definition user can update coord definition in hdfs and
issue a update command. Existing coordinator definition will be replaced by new
definition. The refreshed coordinator
would keep the same coordinator ID, state, and coordinator actions.
User can also provide -dryrun to validate changes.
All created coord action(including in waiting) will use old configuration.
User can rerun actions with -refresh option, -refresh option will use new
configuration to rerun coord action.
Ex.
1. $ oozie job -update <coordID> -config examples/apps/aggregator/job.properties
Will update coord definition and action.
2. $ oozie job -update <coordID>
Will fetch coord definition path from bundle and update coord definition
Diffs (updated)
-----
client/src/main/java/org/apache/oozie/cli/OozieCLI.java e1f551d
client/src/main/java/org/apache/oozie/client/OozieClient.java 46c4288
client/src/main/java/org/apache/oozie/client/rest/JsonTags.java ae391c7
client/src/main/java/org/apache/oozie/client/rest/RestConstants.java a7fe06e
core/pom.xml f0547a3
core/src/main/java/org/apache/oozie/CoordinatorEngine.java 3f10024
core/src/main/java/org/apache/oozie/CoordinatorJobBean.java 5eb134b
core/src/main/java/org/apache/oozie/ErrorCode.java f69d7a2
core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java
9e6a3d5
core/src/main/java/org/apache/oozie/command/coord/CoordUpdateXCommand.java
e69de29
core/src/main/java/org/apache/oozie/executor/jpa/CoordJobQueryExecutor.java
5f7744c
core/src/main/java/org/apache/oozie/servlet/BaseJobServlet.java aa43e68
core/src/main/java/org/apache/oozie/servlet/V0JobServlet.java 443ab6d
core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java ac399e9
core/src/main/java/org/apache/oozie/servlet/V2JobServlet.java 2d26599
core/src/test/java/org/apache/oozie/command/coord/TestCoordUpdateXCommand.java
e69de29
core/src/test/resources/coord-update-test.xml e69de29
docs/src/site/twiki/DG_CommandLineTool.twiki 351f0f2
docs/src/site/twiki/WebServicesAPI.twiki 37c0bc0
pom.xml cb10007
Diff: https://reviews.apache.org/r/20006/diff/
Testing
-------
Invalid xml.
purushah$
/Users/purushah/views/coor_swap/oozie/distro/target/oozie-4.1.0-SNAPSHOT-distro/oozie-4.1.0-SNAPSHOT/bin/oozie
job -oozie http://localhost:11000/oozie -config temp/job.properties -update
0000005-140402104721140-oozie-puru-C -dryrun
Picked up JAVA_TOOL_OPTIONS: -Djava.awt.headless=true
Error: E1023 : E1023: Coord Job update Error: [E0701: XML schema error,
cvc-complex-type.2.4.a: Invalid content was found starting with element
'datasets'. One of '{"uri:oozie:coordinator:0.2":execution,
"uri:oozie:coordinator:0.2":throttle}' is expected.]
purushah$
valid XML.
$ oozie job -oozie http://localhost:11000/oozie -config temp/job.properties
-update 0000005-140402104721140-oozie-puru-C -dryrun
**********Job definition changes**********
@@ -3,8 +3,8 @@
<concurrency>1</concurrency>
</controls>
<input-events>
- <data-in name="input" dataset="raw-logs">
- <dataset name="raw-logs" frequency="20"
initial-instance="2010-01-01T00:00Z" timezone="UTC" freq_timeunit="MINUTE"
end_of_duration="NONE">
+ <data-in name="input" dataset="raw-logs-rename">
+ <dataset name="raw-logs-rename" frequency="20"
initial-instance="2010-01-01T00:00Z" timezone="UTC" freq_timeunit="MINUTE"
end_of_duration="NONE">
<uri-template>hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/</uri-template>
<done-flag />
</dataset>
**********************************
**********Job conf changes**********
@@ -8,10 +8,6 @@
<value>hdfs://localhost:9000/user/purushah/examples/apps/aggregator/coordinator.xml</value>
</property>
<property>
- <name>aaaaaa</name>
- <value>b</value>
- </property>
- <property>
<name>user.name</name>
<value>purushah</value>
</property>
@@ -28,6 +24,10 @@
<value>hdfs://localhost:9000</value>
</property>
<property>
+ <name>adding</name>
+ <value>new</value>
+ </property>
+ <property>
<name>jobTracker</name>
<value>localhost:9001</value>
</property>
**********************************
$
$
/Users/purushah/views/coor_swap/oozie/distro/target/oozie-4.1.0-SNAPSHOT-distro/oozie-4.1.0-SNAPSHOT/bin/oozie
job -oozie http://localhost:11000/oozie -config temp/job.properties -update
0000000-140408140947673-oozie-puru-C
Picked up JAVA_TOOL_OPTIONS: -Djava.awt.headless=true
Error: E0803 : E0803: IO error, E1023: Coord Job update Error: [Frequency can't
be changes. Old frqenecy 60 new frequency 180]
$
Tested Doc.........
DG_CommandLineTool.html#Updating_coordinator_definition_and_properties
Updating coordinator definition and properties
Existing coordinator definition will be replaced by new definition. The
refreshed coordinator would keep the same coordinator ID, state, and
coordinator actions. All created coord action(including in WAITING) will use
old configuration. One can rerun actions with -refresh option, -refresh option
will use new configuration to rerun coord action
Update command also verifies coordinator definition like submit command, if
there is any issue with definition, update will fail. Update command with
-dryrun will show coordinator definition and properties differences. Config
option is optional, if not specified existing coordinator property is used to
find coordinator path.
Update command doesn't update frequency, end time, start time, timzone, name,
execution order. Update command will fail if any of this element has changed in
job definition.
$ oozie job -oozie http://localhost:11000/oozie -config job.properties -update
0000005-140402104721140-oozie-puru-C -dryrun**********Job definition
changes**********
@@ -3,8 +3,8 @@
<concurrency>1</concurrency>
</controls>
<input-events>
- <data-in name="input" dataset="raw-logs">
- <dataset name="raw-logs" frequency="20"
initial-instance="2010-01-01T00:00Z" timezone="UTC" freq_timeunit="MINUTE"
end_of_duration="NONE">
+ <data-in name="input" dataset="raw-logs-rename">
+ <dataset name="raw-logs-rename" frequency="20"
initial-instance="2010-01-01T00:00Z" timezone="UTC" freq_timeunit="MINUTE"
end_of_duration="NONE">
<uri-template>hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/</uri-template>
<done-flag />
</dataset>
**********************************
**********Job conf changes**********
@@ -8,10 +8,6 @@
<value>hdfs://localhost:9000/user/purushah/examples/apps/aggregator/coordinator.xml</value>
</property>
<property>
- <name>old</name>
- <value>test</value>
- </property>
- <property>
<name>user.name</name>
<value>purushah</value>
</property>
@@ -28,6 +24,10 @@
<value>hdfs://localhost:9000</value>
</property>
<property>
+ <name>adding</name>
+ <value>new</value>
+ </property>
+ <property>
<name>jobTracker</name>
<value>localhost:9001</value>
</property>
**********************************
WebServicesAPI.html#Updating_coordinator_definition_and_properties
Existing coordinator definition and properties will be replaced by new
definition and properties. Refer Updating coordinator definition and properties
PUT oozie/v2/job/0000000-140414102048137-oozie-puru-C?action=update
Response:
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
{"update":
{"diff":"**********Job definition
changes**********\n******************************************\n**********Job
conf changes****************\n@@ -8,16 +8,12 @@\n
<value>hdfs:\/\/localhost:9000\/user\/purushah\/examples\/apps\/aggregator\/coordinator.xml<\/value>\r\n
<\/property>\r\n <property>\r\n
- <name>user.name<\/name>\r\n
- <value>purushah<\/value>\r\n
- <\/property>\r\n
- <property>\r\n <name>start<\/name>\r\n
<value>2010-01-01T01:00Z<\/value>\r\n <\/property>\r\n
<property>\r\n
- <name>newproperty<\/name>\r\n
- <value>new<\/value>\r\n
+ <name>user.name<\/name>\r\n
+ <value>purushah<\/value>\r\n <\/property>\r\n <property>\r\n
<name>queueName<\/name>\r\n******************************************\n"
}
}
Thanks,
Purshotam Shah
