> On Aug. 16, 2017, 8:33 p.m., Robert Kanter wrote:
> > client/src/main/resources/hive-action-0.7.xsd
> > Lines 49 (patched)
> > <https://reviews.apache.org/r/61529/diff/2/?file=1796924#file1796924line49>
> >
> > Can we put CONFIGURATION into oozie-common?
> >
> > Same with PREPARE, DELETE, and MKDIR?
>
> Attila Sasvari wrote:
> Right now I changed oozie-common so that its targetNamespace is
> targetNamespace="uri:oozie:workflow:1.0"
> This way common prefix is not needed if ``<launcher>`` is present in the
> global config (include statement can be used).
>
> ```
> <global>
> <launcher>
> <workflow:memory>1024</workflow:memory>
> <workflow:vcores>2</workflow:vcores>
> <workflow:java-opts>-verbose:class</workflow:java-opts>
> <workflow:env>dummyEnv=42</workflow:env>
> <workflow:queue>dummyQueue</workflow:queue>
> </launcher>
> </global>
> ```
>
> Other actions use the import statement and still need the ``workflow``
> prefix for the vcore,memory, etc. if they include the ``<launcher>`` element.
> They also need to define workflow namespace with ``xmlns:workflow``.
>
> ```
> <workflow-app xmlns="uri:oozie:workflow:1.0"
> xmlns:workflow="uri:oozie:workflow:1.0" name="shell-wf">
> <start to="shell-node"/>
> <action name="shell-node">
> <shell xmlns="uri:oozie:shell-action:1.0">
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <launcher>
> <workflow:memory>1024</workflow:memory>
> <workflow:vcores>2</workflow:vcores>
> <workflow:java-opts>-verbose:class</workflow:java-opts>
> <workflow:env>dummyEnv=42</workflow:env>
> <workflow:queue>dummyQueue</workflow:queue>
> <workflow:sharelib>a,b,c</workflow:sharelib>
> </launcher>
> ```
>
> ``xs:any`` would require the prefix (like the sla complex type). Using
> ``<xs:any namespace="##other" minOccurs="1" maxOccurs="1"/>`` could work, but
> is too permissive - for example you could add any element (e.g. a hive
> action) instead of a ``launcher``.
>
> I am trying to find a better way, but please share if you have some ideas.
>
> Attila Sasvari wrote:
> If I use ``ref`` like ``<xs:element ref="workflow:launcher" minOccurs="0"
> maxOccurs="1"/>`` for example in the hive action, it becomes a bit more clean
> to define a ``launcher`` element:
>
> ```
> <workflow-app xmlns="uri:oozie:workflow:1.0" name="hive-wf">
> ...
> <hive xmlns="uri:oozie:workflow:hive-action:1.0">
> ...
> <launcher xmlns="uri:oozie:workflow:1.0">
> <memory>1024</memory>
> </launcher>"
> ```
>
> This way it is not needed to define the workflow namespace in the
> workflow element and use it as prefix.
>
> I was also considering to add other elements to the common workflow
> namespace. Because of the namespace differences of actions, it would not be
> easy to include say ``configuration`` - we would need to define workflow
> namespace and prefix.
>
> Peter Cseh wrote:
> I like the idea of setting xmlns for launcher inside the launcher tag if
> we don't have to use prefixes after that.
>
> Robert Kanter wrote:
> Have you tried using include instead of import? From some googling, it
> sounds like if you (a) use include, (b) don't set a target namespace, and (c)
> don't have a namespace in the included xsd, then it will work the way we
> want.
>
>
> https://stackoverflow.com/questions/2357943/whats-the-difference-between-xsdinclude-and-xsdimport
> https://stackoverflow.com/questions/914020/xsd-with-imports-and-namespaces
>
> https://stackoverflow.com/questions/9805359/how-to-handle-multiple-namespaces-with-different-uri-in-xsd
>
> This link talks about this case from an XSD spec point of view, but it's
> hard to understand:
> https://www.w3.org/TR/xmlschema-1/#compound-schema
>
>
> If that works, then I'd vote we go for that. Otherwise, I'd vote for
> whatever is easiest for the user. If that means we have to duplicate XSD
> code like we do today, that's ok.
Thanks for the ideas. I have already read some of those entries. Anyway, I
tried again the suggestion: I had to use ``import`` otherwise referencing did
not work:
- Using ``<xs:include schemaLocation="oozie-common-1.0.xsd"/>`` and
``<xs:element ref="launcher" minOccurs="0" maxOccurs="1"/>`` :
```
Caused by: org.xml.sax.SAXParseException; systemId:
file:///Users/asasvari/workspace/apache/oozie_dup/core/oozie-workflow-1.0.xsd;
lineNumber: 71; columnNumber: 69; src-resolve.4.1: Error resolving component
'launcher'. It was detected that 'launcher' has no namespace, but components
with no target namespace are not referenceable from schema document
'file:///Users/asasvari/workspace/apache/oozie_dup/core/oozie-workflow-1.0.xsd'.
If 'launcher' is intended to have a namespace, perhaps a prefix needs to be
provided. If it is intended that 'launcher' has no namespace, then an 'import'
without a "namespace" attribute should be added to
'file:///Users/asasvari/workspace/apache/oozie_dup/core/oozie-workflow-1.0.xsd'.
```
- Using ``<xs:element name="launcher" type="LAUNCHER" minOccurs="0"
maxOccurs="1"/>`` also resulted in:
```
Caused by: org.xml.sax.SAXParseException; systemId:
file:///Users/asasvari/workspace/apache/oozie_dup/core/oozie-workflow-1.0.xsd;
lineNumber: 71; columnNumber: 86; src-resolve.4.1: Error resolving component
'LAUNCHER'. It was detected that 'LAUNCHER' has no namespace, but components
with no target namespace are not referenceable from schema document
'file:///Users/asasvari/workspace/apache/oozie_dup/core/oozie-workflow-1.0.xsd'.
If 'LAUNCHER' is intended to have a namespace, perhaps a prefix needs to be
provided. If it is intended that 'LAUNCHER' has no namespace, then an 'import'
without a "namespace" attribute should be added to
'file:///Users/asasvari/workspace/apache/oozie_dup/core/oozie-workflow-1.0.xsd'.
```
When I switched to ``<xs:import schemaLocation="oozie-common-1.0.xsd"/>`` and
``<xs:element ref="launcher" minOccurs="0" maxOccurs="1"/>``, the following
exception was thrown by Xerces:
```
org.xml.sax.SAXParseException; lineNumber: 4; columnNumber: 24;
cvc-complex-type.2.4.a: Invalid content was found starting with element
'memory.mb'. One of '{memory.mb, vcores, java-opts, env, queue, sharelib}' is
expected.
```
Running the sqoop action validation test was also strange:
```
org.xml.sax.SAXParseException; lineNumber: 3; columnNumber: 19;
cvc-complex-type.2.4.a: Invalid content was found starting with element
'launcher'. One of '{"uri:oozie:workflow:1.0":job-tracker,
"uri:oozie:workflow:1.0":resource-manager, "uri:oozie:workflow:1.0":name-node,
"uri:oozie:workflow:1.0":job-xml, launcher,
"uri:oozie:workflow:1.0":configuration}' is expected.
```
It is strange that referencing an element with a complex type from a default
namespace fails to validate.
- Attila
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review183065
-----------------------------------------------------------
On Sept. 8, 2017, 10:04 a.m., Attila Sasvari wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
>
> (Updated Sept. 8, 2017, 10:04 a.m.)
>
>
> Review request for oozie and Peter Cseh.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> New workflow xsd to allow changing launcher configuration (e.g. setting java
> system properties)
>
>
> Diffs
> -----
>
> client/src/main/resources/hive-action-1.0.xsd PRE-CREATION
> client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION
> client/src/main/resources/shell-action-1.0.xsd PRE-CREATION
> client/src/main/resources/spark-action-1.0.xsd PRE-CREATION
> client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
> bca79aa052521ea4f6f16e76bd69f84fb16be790
> core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java
> 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711
> core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java
> 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a
> core/src/main/java/org/apache/oozie/jms/MessageReceiver.java
> 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7
> core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java
> ffc29af2f834da6d0890ea7215c5a62fd7cd693e
> core/src/main/java/org/apache/oozie/service/SchemaService.java
> 137e2c0b4840c6d3858683db1aa38f54bd55be92
> core/src/main/java/org/apache/oozie/util/WritableUtils.java
> aa027e37ba23d4f481698e0bcd93c26c763a0b1f
> core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION
> core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java
> PRE-CREATION
>
> core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java
> PRE-CREATION
>
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java
> a74e5c759fe3e336e7e98c61b8f5ac52efa4100c
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java
> 23df0867e5984c518e8424d79365997bec28f619
> core/src/main/resources/oozie-default.xml
> 0d174b5069aff36ca3a9d48722c5bba69623a046
>
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java
> d74160a09ee237f4067e00ded526e7ec94528a6b
> core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java
> 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482
>
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java
> ce674adf9cffa20690083e6298875a6f9047a109
> core/src/test/java/org/apache/oozie/service/TestSchemaService.java
> 88a10dbead32879334c3953a90a411a3a18fc0b3
> core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java
> a882c82b25154338f89e8a90f831b6f0300fbfe7
>
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java
> 21332404bea79fe0a3cea045d57b3e52a1f15cae
> core/src/test/resources/wf-schema-global-launcherconf-override.xml
> PRE-CREATION
> core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java
> 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java
> ff999ff2cd4875fbf3ad5e1a7eacf9972e848630
>
>
> Diff: https://reviews.apache.org/r/61529/diff/8/
>
>
> Testing
> -------
>
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that
> verbose class loading information is printed out of stdout of the launcher job
>
>
> Thanks,
>
> Attila Sasvari
>
>
