Steve,
Thank you for your quick response. We will have to give some thought as
to how we
can get the results needed for the XML payload validation. Maybe we can
pull out the element with
RuntimeProperties = “stringAsXML=true” attribute and validate it where
we parse the overall binary
Input.
A few points:
1) The exception generated is not from our code but from the call to
validator.validate(new DOMSource(document));
2) If, when envisioning the embedded XML feature, the validation of the
payload would not take place in the InfosetOutputter, why does the example
provided extend InfossetOutputter?
What am I missing?
Thank you for your time,
Mark T
.
-----Original Message-----
From: Steve Lawrence <[email protected]>
Sent: Friday, May 13, 2022 5:00 AM
To: [email protected]
Subject: EXT :Re: Re: Re: Re: Extending JAPI for processing binary input with
an XML Payload
There isn't currently a way for InfosetOutputter's to add to the diagnostics.
In order to do so, an outputter would need access to the PState (where
diagnostics are stored), which would require an API change. I'm not sure we'd
want to give direct access to the PState though--some discussions would be
needed to figure out what that API would look like to get diagnostics out of a
outputter and into the PState.
Also, note that throwing an Exception, whether it be a SAXException or
something else, won't really give the behavior your want. That exception will
just bubble up and lead to an "Abort" error about an unexpected exception being
thrown.
The one exception you could throw is a SchemaDefinitionError and that will be
added to diagnostics. But it will immediately stop processing and you will not
get a complete infoset. This is different than normal validation that Daffodil
does which happens at the end of processing and does not prevent the creation
of an infoset.
When we first envisioned this embedded XML feature, we imagined that the
validation of the payload would not take place in the InfosetOutputter, but
instead would happen after the parse completed in some other process that is
purely for validation. This process would use a schema that combines the DFDL
schema with the schema for the embedded XML so that it would validate correctly.
If you really wanted to do it all in Daffodil, maybe another option would be to
create a custom Validator. This Validator would need to find/extract/unescape
the payload XML (the XML that the Validator gets is not the same as what the
InfosetOutputter creates), and then you can do whatever validation is
necessary. The Validator API does support returning errors/warnings from
validation and adding them to the diagnostics.
On 5/12/22 9:55 PM, Thompson, Mark M [US] (DS) wrote:
> Hello again from XML Payload world,
> - We have succeeded in implementing an extension of
> XMLInfosetOutputter that will parse and validation a binary file
> with an XML payload. It will validate and report if the
> binary fields do not conform to the DFDL schema facet restrictions.
> - We have also implemented a routine to validate the XML
> payload against a schema which will report if the payload is
> malformed. An exception (SAXException) is generated in this case.
> What is not clear is how one gets the Diagnostics to be
> populated with this info or, if they are populated, how to make
> the Diagnostics available to the routine that called
> DataProcessor.Parse() in the first place. The Diagnostics are available
> here for the validation of the binary inputs against the
> corresponding schema but not those reported for the XML payload.
> Do I need to extend the ErrorHandler class and use the
> warning and/or error methods to populate Diagnostics as part of the
> Exception handling for code below?
> The following is an excerpt of the code that seems to do
> what I want except for making the SAXException info available.
> document = db.parse(new InputSource (new StringReader (xmlPayload)));
> // load a WXS schema, represented by a Schema instance
> StreamSource schemaFile = new StreamSource(new
> File(schemaFileName));
> Schema schema = s_factory.newSchema(schemaFile);
> // create a Validator instance, which can be used
> to validate an instance document
> javax.xml.validation.Validator validator =
> schema.newValidator();
> System.out.println("**** validateXMLPayload - Try
> to validate XML Payload.");
> // validate the DOM tree
> validator.validate(new DOMSource(document));
> If anyone can provide a clue as to what I’m missing or
> invalid assumptions, I would greatly appreciate it.
> Thank you for your time,
> Mark T
> -----Original Message-----
> From: Steve Lawrence <[email protected]<mailto:[email protected]>>
> Sent: Thursday, April 28, 2022 4:36 PM
> To: [email protected]<mailto:[email protected]>
> Subject: EXT :Re: Re: Re: Extending JAPI for processing binary input
> with an XML Payload It looks like our InfosetOutputter API isn't very
> Java friendly, at least once you start needing to get information out
> of the ERD. I'd open a ticket, but it looks like JIRA is down at the
> moment.
> I think the way to fix your issue is to just use straight reference
> equality comparison with NodeInfo.String(), e.g.:
> if (simple.erd().optPrimType().get() == NodeInfo.String()) { ...
> } The instance that NodeInfo.String() returns is essentially a
> singleton to represent DFDL String primitives, which is also what is
> returned from optPrimtType().get().
> I'm not sure why our InfosetOutputter uses isInstanceOf[String.Kind]
> instead of just using reference equality with NodeInfo.String. Might
> have been old code that never got updated...
> On 4/28/22 1:24 PM, Thompson, Mark M [US] (DS) wrote:
>> Hello all,
>>
>> We are attempting to generate a Java class for this XML payload by
>> extending XMLInfosetOutputter using XMLTextInfosetOUtputter
>> As a guide.
>> We're having trouble coming up with a Java equivalent for the
>> StartSimple() method code:
>> if
>> (simple.erd.optPrimType.get.isInstanceOf[NodeInfo.String.Kind]) {
>>
>> We've tried the following with no luck:
>> if (simple.erd().optPrimType().get() instanceof
>> NodeInfo.String().Kind) {
>> and
>> if
>> (simple.erd().optPrimType().get().isInstanceOf(NodeInfo.String().???)
>> )
>> {
>>
>> Any input or suggestions are greatly appreciated.
>>
>> Thank you for your time,
>> Mark T
>>
>> -----Original Message-----
>> From: Steve Lawrence <[email protected]
>> <mailto:[email protected]>>
>> Sent: Thursday, April 21, 2022 4:31 AM
>> To: [email protected]<mailto:[email protected]>
>> <mailto:[email protected]>
>> Subject: EXT :Re: Re: Extending JAPI for processing binary input with
>> an XML Payload
>>
>> > Contributing to Daffodil is likely not an option for my company.
>>
>> That's sad to hear, and I hope you can try to persuade your company to
>> change this position. Opensource works best with many contributors from many
>> different organizations. And if your company depends on Daffodil,
>> contributing back to it can help ensure its success and continued growth.
>>
>> > Based on my limited understanding, this seems to suggest that I
>> would have to add another infoset type to make use of the new class
>>
>> If you're using the CLI, this is correct. Infoset inputters and
>> outputters must currently be hardcoded in. If you're using the API to
>> invoke Daffodil this isn't a problem, but if you're using the CLI
>> then you must modify the CLI to support your new classes. This
>> hasn't been an issue so far since anytime a new one is needed we just
> added it to Daffodil.
>>
>> Part of the reason for this is that creating InfosetInputters/Outputters and
>> getting the result is very specific to the infoset representation.
>> It's probably possible that the API could be made more generic, and then we
>> could use a ServiceLoader or something to support pluggable infoset
>> inputters/outputters, but that hasn't be a priority so far.
>>
>> On 4/20/22 10:08 PM, Thompson, Mark M [US] (DS) wrote:
>>> Hi Steve,
>>>
>>> I have been looking into your suggestions and researching the
>>> existing code and here
>>> Is where I'm at:
>>> - Contributing to Daffodil is likely not an option for my company.
>>> - I've been looking into XMLTextInfosetInputter and
>>> ScalaXMLInfosetOutputter for insight.
>>> What I don't understand (Java and Scala newby) is, having extended
>>> InfosetOutputter with a new class,
>>> the Daffodil parse operation magically has access to this new
>>> class. I see, in main.scala, where
>>> XMLTextInfosetOutputter and ScalaXMLInfosetOutputter are
>>> instantiated based on command line
>>> Input for infoset type. Based on my limited understanding, this
>>> seems to suggest that I would have
>>> To add another infoset type to make use of the new class.
>>>
>>> Can someone clear up my misunderstanding or point to some info that
>>> will cause the scales to fall
>>> From my eyes?
>>>
>>> Thank you for your time,
>>> Mark T
>>>
>>> -----Original Message-----
>>> From: Steve Lawrence <[email protected]
>>> <mailto:[email protected]>>
>>> Sent: Friday, April 8, 2022 6:31 AM
>>> To: [email protected]<mailto:[email protected]>
>>> <mailto:[email protected]>
>>> Subject: EXT :Re: Extending JAPI for processing binary input with an
>>> XML Payload
>>>
>>> I've put a decent amount of thought into this issue, I'll add some
>>> additional info that might help out...
>>>
>>> > for example, the startSimple() method. It would be helpful to have an
>>> idea what a given method does normally when there is no class extension.
>>>
>>> The InfosetOutputter class is an abstract class with no default
>>> implementations, so an implementation of all functions, including
>>> startSimple, is required--InfosetInputters don't do anything by default.
>>> The InfosetOutputters that are bundled with Daffodil implement all of these
>>> functions to create the kind of infoset that is output (e.g. XML text,
>>> Scala Nodes, JSON). If you want to take a look at how our bundled
>>> implementations handle their logic, they are all in this folder:
>>>
>>> https://github.com/apache/daffodil/tree/main/daffodil-runtime1/src/m
>>> a
> <https://github.com/apache/daffodil/tree/main/daffodil-runtime1/src/ma
> >
>>> i
>>> n/scala/org/apache/daffodil/infoset
>>>
>>> If you plan to keep your InfosetOutputter outside of the Daffodil codebase,
>>> you'll need to implement all of the functions/logic yourself.
>>> Our current implementions are mostly private, so extending and overriding
>>> them to do you you want is likely not possible.
>>>
>>> A good place to start would be to pick one that that's close to what you
>>> want and modify the various functions as described in the example
>>> implementation in the runtime properties proposal:
>>>
>>> https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Run
>>> t
> <https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Runt
> >
>>> i
>>> me+Properties
>>>
>>> Alternatively, it might be easier to just add the support to our
>>> existing bundled InfosetOutputters to support this capability, and
>>> contribute that work to Daffodil. Not only will this probably be
>>> easier since you can reuse alot of the existing logic, but our
>>> ability to provide support greatly increases when it's related to
> things being added to Daffodil. This feature is a request we've seen
> multiple times--it would be a very welcome addition and we'd be happy
> help get it added to Daffodil.
>>>
>>>
>>>
>>> On 4/8/22 8:42 AM, Mike Beckerle wrote:
>>>> Mark,
>>>>
>>>> There definitely are javadoc jars, and the javadoc is also online
>>>> here (for release 3.3.0)
>>>> https://daffodil.apache.org/docs/latest/javadoc/
> <https://daffodil.apache.org/docs/latest/javadoc/>
>>>> <https://daffodil.apache.org/docs/latest/javadoc/
> <https://daffodil.apache.org/docs/latest/javadoc/>>
>>>>
>>>> The source and javadoc jars are normally pulled down using maven or
>>>> sbt. But you can find them directly on maven central also.
>>>>
>>>> For example for the daffodil-runtime1 jar, all the jars, compiled
>>>> binary, sources, and the javadoc jar is at
>>>> https://repo1.maven.org/maven2/org/apache/daffodil/daffodil-runtime
>>>> 1
> <https://repo1.maven.org/maven2/org/apache/daffodil/daffodil-runtime1>
>>>> _
>>>> 2
>>>> .12/3.3.0/
>>>> <https://repo1.maven.org/maven2/org/apache/daffodil/daffodil-runtim
>>>> e
>>>> 1
>>>> _
>>>> 2.12/3.3.0/>
>>>>
>>>>
>>>> On Thu, Apr 7, 2022 at 10:07 PM Thompson, Mark M [US] (DS)
>>>> <[email protected] <mailto:[email protected]
> <mailto:[email protected] <mailto:[email protected]>>> wrote:
>>>>
>>>> Greetings to All,____
>>>>
>>>> __ __
>>>>
>>>> Caveat: I am new to Daffodil and DFDL. I’m familiar
>>>> with the
>>>> basic functionality and have successfully written____
>>>>
>>>> DFDL schemas to handle text and
>>>> binary inputs
>>>> to the parse/unparse cycle (separate assignments). ____
>>>>
>>>> __ __
>>>>
>>>> I am working on a task that requires the processing of binary files
>>>> containing an XML payload. The goal is to be____
>>>>
>>>> able to validate the raw (un-escapified) XML against
>>>> an XML
>>>> schema during the parsing phase. I have been researching____
>>>>
>>>> the Daffodil JAPI InfosetOutputter family of classes
>>>> as a
>>>> first step. I’ve also read Steve Lawrence’s write-up on the
>>>> ____
>>>>
>>>> RuntimePropertiies proposal. What I have yet to
>>>> grasp or find
>>>> is what InfosetOutputter functionality is to be extended
>>>> .____
>>>>
>>>> I’ve searched high and low but I have yet to find any
>>>> detailed information on what basic functionality is provided
>>>> by,____
>>>>
>>>> for example, the startSimple() method. It would be helpful to have
>>>> an idea
>>>> what a given method does normally when____
>>>>
>>>> there is no class extension.____
>>>>
>>>> __ __
>>>>
>>>> Can anyone recommend a place to start or a source of info on the
>>>> InfosetOutputter family of classes?____
>>>>
>>>> __ __
>>>>
>>>> I am working in NetBeans and every class I find seems to
>>>> indicate that:____
>>>>
>>>> Javadoc not found. Either Javadoc documentation for this item does
>>>> not exist
>>>> or ____
>>>>
>>>> you have not added specified Javadoc in the
>>>> Java
>>>> Platform Manager or the Library ____
>>>>
>>>> Manager.____
>>>>
>>>> __ __
>>>>
>>>> Are there Javadoc(s) available corresponding to the
>>>> Daffodil
>>>> JAPI classes?____
>>>>
>>>> __ __
>>>>
>>>> It feels like I’m missing some basic configuration
>>>> setup
>>>> settings or files. Any guidance or suggestions will____
>>>>
>>>> be greatly appreciated.____
>>>>
>>>> __ __
>>>>
>>>> Thank you for your time,____
>>>>
>>>> Mark M. Thompson____
>>>>
>>>> Northrop Grumman Defense Systems____
>>>>
>>>> Software Engineer____
>>>>
>>>> (818) 712-7439____
>>>>
>>>> __ __
>>>>
