Steve,
Thank you for your quick response. My JVM was set to the default max
(1024 M). I changed it to 4 Gb
and, while it took a long time to finish, the parse completed with the
appropriate errors. The field was
appropriately constrained per requirement but that value exceeded the
JVM memory Max prior to
the change. The Daffodil version in use is 3.2.1.
Thank you for your time,
Mark T
-----Original Message-----
From: Steve Lawrence <[email protected]>
Sent: Thursday, August 4, 2022 5:16 AM
To: [email protected]
Subject: EXT :Re: Daffodil 3.2.1 - An Unexpected exception occurred...
Thanks for reporting. How much memory have you given to your JVM? And what
version of Daffodil are you using?
It looks like the deliberate junk value you reference is about 1.5 GB.
So Daffodil will try to create a 1.5GB array to store the hex binary, and if
you don't have enough memory it will result in the OOM exception.
There are a couple of solutions here:
1) Put an assert on the length field to ensure it is a reasonable size.
For example:
<xs:element name="MDO_MovieDataSize" type="xs:int" ...>
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/";>
<dfdl:assert>{ . le 1000000 }</dfdl:assert>
</xs:appinfo>
</xs:annotation>
</xs:element>
2) Similar to above, put an xs:restriction on the length field to ensure it's a
reasonable size, add an assert to check that restriction:
<xs:element name="MDO_MovieDataSize" ...>
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/";>
<dfdl:assert>{ dfdl:checkConstraints(.) }</dfdl:assert>
</xs:appinfo>
</xs:annotation>
<xs:simpleType>
<xs:restriction base="xs:int">
<xs:maxInclusive value="1000000" />
</xs:restriction>
</xs:simpleType>
</xs:element>
3) Set the "maxHexBinaryLengthInBytes" tunable. This will create a processing
error if the length of a hexBinary field is larger than that tunable. This
value defaults to 2GB, but could be set to a lower value if you know your hex
binary will never be that large.
- Steve
On 8/3/22 6:40 PM, Thompson, Mark M [US] (DS) wrote:
> All,
>
> I am reporting the occurrence of an exception as requested. The
> attached file contains a trace of the Exception with
>
> what I believe the relevant info.
>
> * Command executed
>
> daffodil -t -vv parse -s ..\..\..\MPRemote.dfdl.xsd -V limited -o
> image_file_name-8.invalid.dfdl.xml -r MP_REMOTE_Fields
> image_file_name-8.invalid
>
> * It appears that Daffodil does not like large values (deliberate junk in
> this
> case) when used as a size for an xs:hexBinary element. See
>
> <MDO_MovieDataSize> and <MDO_MovieData> respectively in the attached
> trace. In this case, there is nowhere near that much data available
>
> In the test binary input. Normally, in the case of insufficient data,
> Daffodil errors out gracefully and indicates that there was insufficient dat.
>
> Overview:
>
> * I am not at liberty to provide the actual schema files.
> * I may be able to provide test messages if necessary. I’m hoping that the
> attached trace provides more than enough info.
> * The Input test files to Daffodil are binary.
> * The command used in this case is listed above.
> * Command line options: -t -vv parse -V limited
> * Daffodil version: 3.2.1
> * Offending element: <MDO_MovieData>
>
> Thank you for your time,
>
> Mark M. Thompson
>
> Northrop Grumman Defense Systems
>
> Software Engineer
>
> (818) 712-7439
>
