> On Aug. 16, 2017, 8:33 p.m., Robert Kanter wrote: > > client/src/main/resources/hive-action-0.7.xsd > > Lines 49 (patched) > > <https://reviews.apache.org/r/61529/diff/2/?file=1796924#file1796924line49> > > > > Can we put CONFIGURATION into oozie-common? > > > > Same with PREPARE, DELETE, and MKDIR? > > Attila Sasvari wrote: > Right now I changed oozie-common so that its targetNamespace is > targetNamespace="uri:oozie:workflow:1.0" > This way common prefix is not needed if ``<launcher>`` is present in the > global config (include statement can be used). > > ``` > <global> > <launcher> > <workflow:memory>1024</workflow:memory> > <workflow:vcores>2</workflow:vcores> > <workflow:java-opts>-verbose:class</workflow:java-opts> > <workflow:env>dummyEnv=42</workflow:env> > <workflow:queue>dummyQueue</workflow:queue> > </launcher> > </global> > ``` > > Other actions use the import statement and still need the ``workflow`` > prefix for the vcore,memory, etc. if they include the ``<launcher>`` element. > They also need to define workflow namespace with ``xmlns:workflow``. > > ``` > <workflow-app xmlns="uri:oozie:workflow:1.0" > xmlns:workflow="uri:oozie:workflow:1.0" name="shell-wf"> > <start to="shell-node"/> > <action name="shell-node"> > <shell xmlns="uri:oozie:shell-action:1.0"> > <job-tracker>${jobTracker}</job-tracker> > <name-node>${nameNode}</name-node> > <launcher> > <workflow:memory>1024</workflow:memory> > <workflow:vcores>2</workflow:vcores> > <workflow:java-opts>-verbose:class</workflow:java-opts> > <workflow:env>dummyEnv=42</workflow:env> > <workflow:queue>dummyQueue</workflow:queue> > <workflow:sharelib>a,b,c</workflow:sharelib> > </launcher> > ``` > > ``xs:any`` would require the prefix (like the sla complex type). Using > ``<xs:any namespace="##other" minOccurs="1" maxOccurs="1"/>`` could work, but > is too permissive - for example you could add any element (e.g. a hive > action) instead of a ``launcher``. > > I am trying to find a better way, but please share if you have some ideas. > > Attila Sasvari wrote: > If I use ``ref`` like ``<xs:element ref="workflow:launcher" minOccurs="0" > maxOccurs="1"/>`` for example in the hive action, it becomes a bit more clean > to define a ``launcher`` element: > > ``` > <workflow-app xmlns="uri:oozie:workflow:1.0" name="hive-wf"> > ... > <hive xmlns="uri:oozie:workflow:hive-action:1.0"> > ... > <launcher xmlns="uri:oozie:workflow:1.0"> > <memory>1024</memory> > </launcher>" > ``` > > This way it is not needed to define the workflow namespace in the > workflow element and use it as prefix. > > I was also considering to add other elements to the common workflow > namespace. Because of the namespace differences of actions, it would not be > easy to include say ``configuration`` - we would need to define workflow > namespace and prefix. > > Peter Cseh wrote: > I like the idea of setting xmlns for launcher inside the launcher tag if > we don't have to use prefixes after that. > > Robert Kanter wrote: > Have you tried using include instead of import? From some googling, it > sounds like if you (a) use include, (b) don't set a target namespace, and (c) > don't have a namespace in the included xsd, then it will work the way we > want. > > > https://stackoverflow.com/questions/2357943/whats-the-difference-between-xsdinclude-and-xsdimport > https://stackoverflow.com/questions/914020/xsd-with-imports-and-namespaces > > https://stackoverflow.com/questions/9805359/how-to-handle-multiple-namespaces-with-different-uri-in-xsd > > This link talks about this case from an XSD spec point of view, but it's > hard to understand: > https://www.w3.org/TR/xmlschema-1/#compound-schema > > > If that works, then I'd vote we go for that. Otherwise, I'd vote for > whatever is easiest for the user. If that means we have to duplicate XSD > code like we do today, that's ok. > > Attila Sasvari wrote: > Thanks for the ideas. I have already read some of those entries. Anyway, > I tried again the suggestion: I had to use ``import`` otherwise referencing > did not work: > - Using ``<xs:include schemaLocation="oozie-common-1.0.xsd"/>`` and > ``<xs:element ref="launcher" minOccurs="0" maxOccurs="1"/>`` : > ``` > Caused by: org.xml.sax.SAXParseException; systemId: > file:///Users/asasvari/workspace/apache/oozie_dup/core/oozie-workflow-1.0.xsd; > lineNumber: 71; columnNumber: 69; src-resolve.4.1: Error resolving component > 'launcher'. It was detected that 'launcher' has no namespace, but components > with no target namespace are not referenceable from schema document > 'file:///Users/asasvari/workspace/apache/oozie_dup/core/oozie-workflow-1.0.xsd'. > If 'launcher' is intended to have a namespace, perhaps a prefix needs to be > provided. If it is intended that 'launcher' has no namespace, then an > 'import' without a "namespace" attribute should be added to > 'file:///Users/asasvari/workspace/apache/oozie_dup/core/oozie-workflow-1.0.xsd'. > > ``` > - Using ``<xs:element name="launcher" type="LAUNCHER" minOccurs="0" > maxOccurs="1"/>`` also resulted in: > ``` > Caused by: org.xml.sax.SAXParseException; systemId: > file:///Users/asasvari/workspace/apache/oozie_dup/core/oozie-workflow-1.0.xsd; > lineNumber: 71; columnNumber: 86; src-resolve.4.1: Error resolving component > 'LAUNCHER'. It was detected that 'LAUNCHER' has no namespace, but components > with no target namespace are not referenceable from schema document > 'file:///Users/asasvari/workspace/apache/oozie_dup/core/oozie-workflow-1.0.xsd'. > If 'LAUNCHER' is intended to have a namespace, perhaps a prefix needs to be > provided. If it is intended that 'LAUNCHER' has no namespace, then an > 'import' without a "namespace" attribute should be added to > 'file:///Users/asasvari/workspace/apache/oozie_dup/core/oozie-workflow-1.0.xsd'. > > ``` > > When I switched to ``<xs:import schemaLocation="oozie-common-1.0.xsd"/>`` > and ``<xs:element ref="launcher" minOccurs="0" maxOccurs="1"/>``, the > following exception was thrown by Xerces: > ``` > org.xml.sax.SAXParseException; lineNumber: 4; columnNumber: 24; > cvc-complex-type.2.4.a: Invalid content was found starting with element > 'memory.mb'. One of '{memory.mb, vcores, java-opts, env, queue, sharelib}' is > expected. > ``` > > Running the sqoop action validation test was also strange: > ``` > org.xml.sax.SAXParseException; lineNumber: 3; columnNumber: 19; > cvc-complex-type.2.4.a: Invalid content was found starting with element > 'launcher'. One of '{"uri:oozie:workflow:1.0":job-tracker, > "uri:oozie:workflow:1.0":resource-manager, > "uri:oozie:workflow:1.0":name-node, "uri:oozie:workflow:1.0":job-xml, > launcher, "uri:oozie:workflow:1.0":configuration}' is expected. > ``` > > It is strange that referencing an element with a complex type from a > default namespace fails to validate.
I tried playing around with this, and looked especially at the third stackoverflow link, and I think I got it working. Here's a proof of concept patch that goes on top of your patch. It changes the oozie-common xsd to not have a namespace, and the hive xsd to use the LAUNCHER type from there without a prefix. I ran TestSchemaService#testHiveActionLauncherConfig to verify. Assuming this isn't a fluke, we'd just need to extend this to the other schemas and move other common things to oozie-common like \<configuration>. Please take a look and let me know what you think. ```` diff --git a/client/src/main/resources/hive-action-1.0.xsd b/client/src/main/resources/hive-action-1.0.xsd index 6fc921ae51..de6e7c1529 100644 --- a/client/src/main/resources/hive-action-1.0.xsd +++ b/client/src/main/resources/hive-action-1.0.xsd @@ -18,11 +18,10 @@ --> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; xmlns:hive="uri:oozie:hive-action:1.0" - xmlns:workflow="uri:oozie:workflow:1.0" elementFormDefault="qualified" targetNamespace="uri:oozie:hive-action:1.0"> - <xs:import namespace="uri:oozie:workflow:1.0"/> + <xs:include schemaLocation="oozie-common-1.0.xsd"/> <xs:element name="hive" type="hive:ACTION"/> @@ -35,7 +34,7 @@ <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/> <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> - <xs:element ref="workflow:launcher" minOccurs="0" maxOccurs="1"/> + <xs:element name="launcher" type="hive:LAUNCHER" minOccurs="0" maxOccurs="1"/> <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/> <xs:choice minOccurs="1" maxOccurs="1"> <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/> diff --git a/client/src/main/resources/oozie-common-1.0.xsd b/client/src/main/resources/oozie-common-1.0.xsd index e149f6d4b6..beb9d4d0ca 100644 --- a/client/src/main/resources/oozie-common-1.0.xsd +++ b/client/src/main/resources/oozie-common-1.0.xsd @@ -17,8 +17,8 @@ limitations under the License. --> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; xmlns:workflow="uri:oozie:workflow:1.0" - elementFormDefault="qualified" targetNamespace="uri:oozie:workflow:1.0"> - <xs:element name="launcher" type="workflow:LAUNCHER"/> + elementFormDefault="qualified"> + <xs:element name="launcher" type="LAUNCHER"/> <xs:complexType name="LAUNCHER"> <xs:choice maxOccurs="unbounded"> diff --git a/core/src/test/java/org/apache/oozie/service/TestSchemaService.java b/core/src/test/java/org/apache/oozie/service/TestSchemaService.java index 79c688e331..d7465cade4 100644 --- a/core/src/test/java/org/apache/oozie/service/TestSchemaService.java +++ b/core/src/test/java/org/apache/oozie/service/TestSchemaService.java @@ -128,7 +128,7 @@ public class TestSchemaService extends XTestCase { "</workflow-app>\n"; private static final String HIVE_ACTION_LAUNCHER_CONF = - "<workflow-app xmlns:workflow=\"uri:oozie:workflow:1.0\" xmlns=\"uri:oozie:workflow:1.0\" name=\"hive-wf\">\n" + + "<workflow-app xmlns=\"uri:oozie:workflow:1.0\" name=\"hive-wf\">\n" + " <start to=\"hive-node\"/>\n" + "\n" + " <action name=\"hive-node\">\n" + @@ -139,7 +139,7 @@ public class TestSchemaService extends XTestCase { " <delete path=\"${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/hive\"/>\n" + " <mkdir path=\"${nameNode}/user/${wf:user()}/${examplesRoot}/output-data\"/>\n" + " </prepare>\n" + - " <launcher xmlns=\"uri:oozie:workflow:1.0\">\n" + + " <launcher>\n" + " <memory.mb>1024</memory.mb>\n" + " <vcores>2</vcores>\n" + " <java-opts>dummyJavaOpts</java-opts>\n" + ```` - Robert ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61529/#review183065 ----------------------------------------------------------- On Sept. 8, 2017, 10:04 a.m., Attila Sasvari wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61529/ > ----------------------------------------------------------- > > (Updated Sept. 8, 2017, 10:04 a.m.) > > > Review request for oozie and Peter Cseh. > > > Repository: oozie-git > > > Description > ------- > > New workflow xsd to allow changing launcher configuration (e.g. setting java > system properties) > > > Diffs > ----- > > client/src/main/resources/hive-action-1.0.xsd PRE-CREATION > client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION > client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION > client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION > client/src/main/resources/shell-action-1.0.xsd PRE-CREATION > client/src/main/resources/spark-action-1.0.xsd PRE-CREATION > client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java > bca79aa052521ea4f6f16e76bd69f84fb16be790 > core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java > 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711 > core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java > 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a > core/src/main/java/org/apache/oozie/jms/MessageReceiver.java > 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7 > core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java > ffc29af2f834da6d0890ea7215c5a62fd7cd693e > core/src/main/java/org/apache/oozie/service/SchemaService.java > 137e2c0b4840c6d3858683db1aa38f54bd55be92 > core/src/main/java/org/apache/oozie/util/WritableUtils.java > aa027e37ba23d4f481698e0bcd93c26c763a0b1f > core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION > core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java > PRE-CREATION > > core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java > PRE-CREATION > > core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java > a74e5c759fe3e336e7e98c61b8f5ac52efa4100c > core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java > 23df0867e5984c518e8424d79365997bec28f619 > core/src/main/resources/oozie-default.xml > 0d174b5069aff36ca3a9d48722c5bba69623a046 > > core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java > d74160a09ee237f4067e00ded526e7ec94528a6b > core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java > 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482 > > core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java > ce674adf9cffa20690083e6298875a6f9047a109 > core/src/test/java/org/apache/oozie/service/TestSchemaService.java > 88a10dbead32879334c3953a90a411a3a18fc0b3 > core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java > a882c82b25154338f89e8a90f831b6f0300fbfe7 > > core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java > 21332404bea79fe0a3cea045d57b3e52a1f15cae > core/src/test/resources/wf-schema-global-launcherconf-override.xml > PRE-CREATION > core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java > 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d > sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java > ff999ff2cd4875fbf3ad5e1a7eacf9972e848630 > > > Diff: https://reviews.apache.org/r/61529/diff/8/ > > > Testing > ------- > > - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed > - java-main example on pseudo hadoop succeeded, modified workflow xml so that > verbose class loading information is printed out of stdout of the launcher job > > > Thanks, > > Attila Sasvari > >
