Re: Fwd: question re: separatorPosition="infix"

[Attila Horvath]

Keep in mind I'm using Daffodil 2.4.0

I understand and your explanation makes sense; however...

When I remove highlighted DFDL separator attributes on line #33 per
suggestion:...
[image: image.png]
I get identical result:...
[image: image.png]
I.E.  still consumed prior to 'tailing-rec' element processing.

Unexpectedly, the only way to get the 'expected' result is by adding
additional :...
[image: image.png]
You've confirmed my understanding of 'infix'; however, not getting expected
results.

Thx - Attila



On Tue, Apr 19, 2022 at 10:40 AM Steve Lawrence 
wrote:

> Thanks for the provided schemas. This helps quite a bit.
>
> In this case, the "record" element does not consume the final CR CR LF
> because it is an infix separator, as you expect.
>
> However, you have an CR CR LF separator on the outer sequence on line
> 33, so you have something like this:
>
> dfdl:separatorPosition="infix">
>  
>
>  
>  
>
>
> So this expects a CR CR LF in between your MU16Type element and a
> trailing-rec element. That is what actually consumes the final CR CR LF
> and then the trailing-rec element happily consumes no data and you get
> the NIL element.
>
> So the suggestion if you want the CRCRLF element to be parsed in the
> tailing-rec is to remove the CR CR LF separator from the outer sequence.
> Note that things get a bit tricky if you do that. In that case, the
> final trailing CR CR LF is not consumed as an infix separator, as
> expected. And then we do get a single trailing-rec element that consumes
> the CR CR LF and we get the  element. But then because
> trailing-rec is unbounded, and happy to try to parse more trailing-rec
> elements. But a trailing-rec element can consume no data because of the
> NIL element. This would mean Daffodil could get stuck in an infinite
> loop happily consuming no data. Daffodil detects this and errors with a
> "No forward progress" message. Simply removing the NIL element from the
> TailType-TLE fixes this issue, this way it's not possible for
> TailType-TLE to consume no data.
>
> - Steve
>
>
> On 4/19/22 10:12 AM, Attila Horvath wrote:
> > ALCON
> >
> > Same question/issue - see attached samp schema/data.
> >
> > Thx - Attila
> >
> > On Tue, Apr 19, 2022 at 10:03 AM http://apache.org>>
> wrote:
> >
> >
> >   >  I can't reproduce this with simpler examples. Can you
> provide your
> >   >  actual schema/data?
> >   >
> >
>
>

dev/users list accept attachment(s) inconsistently

[Attila Horvath]

ALCON

List messages strip formatting: *BOLD*, RED, *ITALICS*, *underlined*, etc...

Orig. message missing attachment on dev list, on follow up message 1 of 2
attachments lost:
[image: image.png]

Re: Fwd: question re: separatorPosition="infix"

[Attila Horvath]

ALCON

Same question/issue - see attached samp schema/data.

Thx - Attila

On Tue, Apr 19, 2022 at 10:03 AM  wrote:

>
> >  I can't reproduce this with simpler examples. Can you provide your
> >  actual schema/data?
> >
>
>
<>

question re: separatorPosition="infix"

[Attila Horvath]

"... Since ‘*separatorPosition*’ is ‘*infix*’, I would have anticipated the
last  record terminator to have been leftover ..."

Please refer to attached PDF.

Thx - Attila

2.4 unparse "bug"

[Attila Horvath]

ALCON

I suspect this unparse bug w/ Daffodil 2.4 has since been fixed. Parsing is
successful. Details as follows:

 daffodil unparse --validate=on -s ... -r ... -o ...

!!
!!   An unexpected exception occurred. This is a bug!   !!
!!

 Please report this bug and help us fix it:

  https://daffodil.apache.org/community/#issue-tracker

 Please include the following exception, the command you
 ran, and any input, schema, or tdml files used that led
 to this bug.


org.apache.daffodil.exceptions.Abort: Invariant broken. Single suspended
expression making no forward progress.
SuppressableSeparatorUnparserSuspendableOperation for Miscellanea
org.apache.daffodil.exceptions.Assert$.abort(Assert.scala:129)
org.apache.daffodil.exceptions.Assert$.invariantFailed(Assert.scala:175)
org.apache.daffodil.processors.unparsers.UStateMain.evalSuspensions(UState.scala:603)
org.apache.daffodil.processors.DataProcessor.unparse(DataProcessor.scala:305)
at org.apache.daffodil.exceptions.Assert$.abort(Assert.scala:129)
at
org.apache.daffodil.exceptions.Assert$.invariantFailed(Assert.scala:175)
at
org.apache.daffodil.processors.unparsers.UStateMain.evalSuspensions(UState.scala:603)
at
org.apache.daffodil.processors.DataProcessor.unparse(DataProcessor.scala:305)
at org.apache.daffodil.Main$.$anonfun$run$15(Main.scala:1135)
at org.apache.daffodil.util.Timer$.getTimeResult(Timer.scala:76)
at org.apache.daffodil.util.Timer$.getResult(Timer.scala:35)
at org.apache.daffodil.Main$.run(Main.scala:1135)
at org.apache.daffodil.Main$.main(Main.scala:1363)
at org.apache.daffodil.Main.main(Main.scala)

'Miscellanea' referenced above is simply:













It occurs when encountering an empty line/record when using
"dfdl:separatorPosition="postfix"



:::



Let me know if more information required.

Thx - Attila

Re: string to integer conversion w/ leading blanks fails

[Attila Horvath]

ALCON

 Appreciate the following suggested workaround:



The snippet above on implementation does work but doesn’t yield lossless
unparse results which is still our goal.

After [much] trial/error I came up w/ alternative approach which deals with
leading whitespace/zeros but produces lossless parse/unparsed results:
[image: image.png]

The snipped above 1st instantiates an element of leading whitespace(s) on
line #134.

Given the satellite number is fixed length of 5 characters, the hidden
group isolates the numerical digits based on the number of whitespace
characters and instantiates an element ‘satellite-num-range’ of variable
length.

This allows lossless string processing and the ability to convert string to
unsignedInt and avoid conversion error due to leading whitespace(s).



Attila


PS: if you can't see embedded image above, see attached PDF.


On Wed, Mar 16, 2022 at 12:21 PM Mike Beckerle  wrote:

> Ok, I found the attachment. Sorry for the delay.
>
> The challenge here is you are thinking the
> xs:unsignedInt(../Line1.02-Satellite) call will tolerate whitespace. Which
> it seems they do not.
>
> I think this is a Daffodil bug, as the constructors like xs:unsignedInt are
> supposed to work like they do in XPath, and the XPath functions spec says
> when converting from strings, that whitespace normalization applies - which
> trims all leading and trailing whitespace. It's less clear about whether
> interior whitespace is collapsed, but definitely leading/trailing seem to
> be trimmed.
>
> So I'll add a JIRA ticket about this.
>
> For how to work around, I suggest parsing the satellite field not as a
> string, but as an unsignedInt from the start.
>
> So like:
>
>  dfdl:lengthKind="explicit" dfdl:length="5"
>   dfdl:textTrimKind="padChar" dfdl:textPadKind="padChar"
> dfdl:textNumberPadCharacter="%SP;" dfdl:textNumberJustification="right"
>   dfdl:textNumberPattern="0"/>
>
> I didn't run this, but I think this will remove leading spaces, and add
> leading spaces to your 5 character element.
>
> Another way to express this, since you need only leading padding is this:
>
>  dfdl:lengthKind="explicit" dfdl:length="5"
>   dfdl:textNumberPattern="* 0"/>
>
> In that textNumberPattern the "* " means spaces are the pad character to be
> used, and when there is no digit for the position of a "#" then the pad
> character from the pattern (not the textNumberPadCharacter) is used.
>
> Both kinds of padding can be used together E.g., so you could have number
> text right justified in a fixed-length field of width 6, using "*" to pad
> to width 5 so that you can get " **123".
>
>  dfdl:lengthKind="explicit" dfdl:length="6"
>   dfdl:textTrimKind="padChar" dfdl:textPadKind="padChar"
> dfdl:textNumberPadCharacter="%SP;" dfdl:textNumberJustification="right"
>   dfdl:textNumberPattern="* 0"/>
>
> I didn't run these, but this is, I believe, how it is supposed to work.
>
>
>
> On Tue, Mar 15, 2022 at 5:23 PM Attila Horvath  >
> wrote:
>
> > Attachment can be found on
> > https://lists.apache.org/list.html?us...@daffodil.apache.org list.
> > Not sure why it didn't show up on dev - I sent msg to both lists.
> >
> > On Tue, Mar 15, 2022 at 3:41 PM Mike Beckerle 
> > wrote:
> >
> > > No attached PDF.
> > >
> > > Removal of blanks from a number would normally be by way of DFDL
> > "padding"
> > > and "trimming" properties.
> > >
> > > You need dfdl:textNumberJustification property, also textTrimKind,
> > > textPadKind, textNumberPadCharacter.
> > >
> > > You can also use textNumberPattern to indicate that a number may have
> > > leading spaces or zeros, but this is mostly about output when those
> > leading
> > > zeros are required.
> > >
> > > Padding on left == Right Justified
> > > Padding on right == Left Justified
> > > Padding around both sides = Center Justified.
> > >
> > >
> > >
> > > On Tue, Mar 15, 2022 at 8:53 AM Attila Horvath <
> > attila.j.horv...@gmail.com
> > > >
> > > wrote:
> > >
> > > > Ping... any assistance appreciated - thx
> > > >
> > > > -- Forwarded message -
> > > > From: Attila Horvath 
> > > > Date: Mon, Mar 14, 2022 at 12:38 PM
> > > > Subject: string to integer conversion w/ leading blanks fails
> > > > To: , 
> > > >
> > > >
> > > > ALCON
> > > >
> > > > Can someone pls suggest a way to convert string to integer if/when
> > > leading
> > > > blanks are present?
> > > >
> > > > My attempts are failing - see attached pdf for more details.
> > > >
> > > > Thx in advance
> > > >
> > > > Attila
> > > >
> > >
> >
>

Re: string to integer conversion w/ leading blanks fails

[Attila Horvath]

Mike

Appreciate the suggested workaround. I did incorporate/test your snippet
per Mar 16, 2022 at 12:21 PM [below] w/ following anticipated results:

[1]Satellite numbers w/ leading whitespace(s) yields lossless unparse
results.
[2]Satellite numbers w/ leading zero(s) or whitespace(s)+zero(s) yield
unparse results that are 'numerically' equivalent |HOWEVER| unparsed target
ASCII file fails to compare w/ parsed source ASCII file due to
<<>> formatting that trims leading
irrelevant characters - see attached parse/source and unparse/target files.
NB "0" formatting yields opposite results by trimming leading
whitspace(s).

The issue/concern of 'lossless' parse/unparse processing for our
organization is fundamental. Our organization has no control over
customers' [legacy] pre-/post- processes |AND| the format of input data.
Ergo lossless end-to-end data transformation is essential b/c if/when
source/target data fail to compare, we're placed in the untenable position
of explaining differences on case by case basis.

The impetus/urgency for my questions below re: ticket is that producing
lossless end-to-end data transformation results via string processing with
suite of XPATH functions is more important/suitable than yielding
numerically 'equivalent' results.

Thx - Attila

On Fri, Mar 18, 2022 at 5:12 AM Attila Horvath 
wrote:

> I know you're preparing to release 3.3.0.
>
> When do think this issue might be resolved? Which point release are you
> targeting?
>
> On a related subject, Daffodil implements a subset of XPATH function.
> Might dev-team consider implementing all XPATH functions in lieu of
> workarounds?
>
> Thx in advance - Attila
>
> On Wed, Mar 16, 2022 at 12:26 PM Mike Beckerle 
> wrote:
>
>> Created https://issues.apache.org/jira/browse/DAFFODIL-2676
>>
>> On Wed, Mar 16, 2022 at 12:21 PM Mike Beckerle 
>> wrote:
>>
>> > Ok, I found the attachment. Sorry for the delay.
>> >
>> > The challenge here is you are thinking the
>> > xs:unsignedInt(../Line1.02-Satellite) call will tolerate whitespace.
>> Which
>> > it seems they do not.
>> >
>> > I think this is a Daffodil bug, as the constructors like xs:unsignedInt
>> > are supposed to work like they do in XPath, and the XPath functions spec
>> > says when converting from strings, that whitespace normalization
>> applies -
>> > which trims all leading and trailing whitespace. It's less clear
>> > about whether interior whitespace is collapsed, but definitely
>> > leading/trailing seem to be trimmed.
>> >
>> > So I'll add a JIRA ticket about this.
>> >
>> > For how to work around, I suggest parsing the satellite field not as a
>> > string, but as an unsignedInt from the start.
>> >
>> > So like:
>> >
>> > > > dfdl:lengthKind="explicit" dfdl:length="5"
>> >   dfdl:textTrimKind="padChar" dfdl:textPadKind="padChar"
>> > dfdl:textNumberPadCharacter="%SP;" dfdl:textNumberJustification="right"
>> >   dfdl:textNumberPattern="0"/>
>> >
>> > I didn't run this, but I think this will remove leading spaces, and add
>> > leading spaces to your 5 character element.
>> >
>> > Another way to express this, since you need only leading padding is
>> this:
>> >
>> > > > dfdl:lengthKind="explicit" dfdl:length="5"
>> >   dfdl:textNumberPattern="* 0"/>
>> >
>> > In that textNumberPattern the "* " means spaces are the pad character to
>> > be used, and when there is no digit for the position of a "#" then the
>> pad
>> > character from the pattern (not the textNumberPadCharacter) is used.
>> >
>> > Both kinds of padding can be used together E.g., so you could have
>> number
>> > text right justified in a fixed-length field of width 6, using "*" to
>> pad
>> > to width 5 so that you can get " **123".
>> >
>> > > > dfdl:lengthKind="explicit" dfdl:length="6"
>> >   dfdl:textTrimKind="padChar" dfdl:textPadKind="padChar"
>> > dfdl:textNumberPadCharacter="%SP;" dfdl:textNumberJustification="right"
>> >   dfdl:textNumberPattern="* 0"/>
>> >
>> > I didn't run these, but this is, I believe, how it is supposed to work.
>> >
>> >
>> >
>>
>>

Re: string to integer conversion w/ leading blanks fails

[Attila Horvath]

I know you're preparing to release 3.3.0.

When do think this issue might be resolved? Which point release are you
targeting?

On a related subject, Daffodil implements a subset of XPATH function.
Might dev-team consider implementing all XPATH functions in lieu of
workarounds?

Thx in advance - Attila

On Wed, Mar 16, 2022 at 12:26 PM Mike Beckerle  wrote:

> Created https://issues.apache.org/jira/browse/DAFFODIL-2676
>
> On Wed, Mar 16, 2022 at 12:21 PM Mike Beckerle 
> wrote:
>
> > Ok, I found the attachment. Sorry for the delay.
> >
> > The challenge here is you are thinking the
> > xs:unsignedInt(../Line1.02-Satellite) call will tolerate whitespace.
> Which
> > it seems they do not.
> >
> > I think this is a Daffodil bug, as the constructors like xs:unsignedInt
> > are supposed to work like they do in XPath, and the XPath functions spec
> > says when converting from strings, that whitespace normalization applies
> -
> > which trims all leading and trailing whitespace. It's less clear
> > about whether interior whitespace is collapsed, but definitely
> > leading/trailing seem to be trimmed.
> >
> > So I'll add a JIRA ticket about this.
> >
> > For how to work around, I suggest parsing the satellite field not as a
> > string, but as an unsignedInt from the start.
> >
> > So like:
> >
> >  > dfdl:lengthKind="explicit" dfdl:length="5"
> >   dfdl:textTrimKind="padChar" dfdl:textPadKind="padChar"
> > dfdl:textNumberPadCharacter="%SP;" dfdl:textNumberJustification="right"
> >   dfdl:textNumberPattern="0"/>
> >
> > I didn't run this, but I think this will remove leading spaces, and add
> > leading spaces to your 5 character element.
> >
> > Another way to express this, since you need only leading padding is this:
> >
> >  > dfdl:lengthKind="explicit" dfdl:length="5"
> >   dfdl:textNumberPattern="* 0"/>
> >
> > In that textNumberPattern the "* " means spaces are the pad character to
> > be used, and when there is no digit for the position of a "#" then the
> pad
> > character from the pattern (not the textNumberPadCharacter) is used.
> >
> > Both kinds of padding can be used together E.g., so you could have number
> > text right justified in a fixed-length field of width 6, using "*" to pad
> > to width 5 so that you can get " **123".
> >
> >  > dfdl:lengthKind="explicit" dfdl:length="6"
> >   dfdl:textTrimKind="padChar" dfdl:textPadKind="padChar"
> > dfdl:textNumberPadCharacter="%SP;" dfdl:textNumberJustification="right"
> >   dfdl:textNumberPattern="* 0"/>
> >
> > I didn't run these, but this is, I believe, how it is supposed to work.
> >
> >
> >
>
>

Re: string to integer conversion w/ leading blanks fails

[Attila Horvath]

Mike et al

See dev/users mail list for attached PDF.

In addition to unexpected  behavior, see unusual behavior
re:
fn:count
dfdl:outputValueCalc ... if (~) then ~ else

Attila

On Wed, Mar 16, 2022 at 12:26 PM Mike Beckerle  wrote:

> Created https://issues.apache.org/jira/browse/DAFFODIL-2676
>
> On Wed, Mar 16, 2022 at 12:21 PM Mike Beckerle 
> wrote:
>
> > Ok, I found the attachment. Sorry for the delay.
> >
> > The challenge here is you are thinking the
> > xs:unsignedInt(../Line1.02-Satellite) call will tolerate whitespace.
> Which
> > it seems they do not.
> >
> > I think this is a Daffodil bug, as the constructors like xs:unsignedInt
> > are supposed to work like they do in XPath, and the XPath functions spec
> > says when converting from strings, that whitespace normalization applies
> -
> > which trims all leading and trailing whitespace. It's less clear
> > about whether interior whitespace is collapsed, but definitely
> > leading/trailing seem to be trimmed.
> >
> > So I'll add a JIRA ticket about this.
> >
> > For how to work around, I suggest parsing the satellite field not as a
> > string, but as an unsignedInt from the start.
> >
> > So like:
> >
> >  > dfdl:lengthKind="explicit" dfdl:length="5"
> >   dfdl:textTrimKind="padChar" dfdl:textPadKind="padChar"
> > dfdl:textNumberPadCharacter="%SP;" dfdl:textNumberJustification="right"
> >   dfdl:textNumberPattern="0"/>
> >
> > I didn't run this, but I think this will remove leading spaces, and add
> > leading spaces to your 5 character element.
> >
> > Another way to express this, since you need only leading padding is this:
> >
> >  > dfdl:lengthKind="explicit" dfdl:length="5"
> >   dfdl:textNumberPattern="* 0"/>
> >
> > In that textNumberPattern the "* " means spaces are the pad character to
> > be used, and when there is no digit for the position of a "#" then the
> pad
> > character from the pattern (not the textNumberPadCharacter) is used.
> >
> > Both kinds of padding can be used together E.g., so you could have number
> > text right justified in a fixed-length field of width 6, using "*" to pad
> > to width 5 so that you can get " **123".
> >
> >  > dfdl:lengthKind="explicit" dfdl:length="6"
> >   dfdl:textTrimKind="padChar" dfdl:textPadKind="padChar"
> > dfdl:textNumberPadCharacter="%SP;" dfdl:textNumberJustification="right"
> >   dfdl:textNumberPattern="* 0"/>
> >
> > I didn't run these, but this is, I believe, how it is supposed to work.
> >
> >
> >
> > On Tue, Mar 15, 2022 at 5:23 PM Attila Horvath <
> attila.j.horv...@gmail.com>
> > wrote:
> >
> >> Attachment can be found on
> >> https://lists.apache.org/list.html?us...@daffodil.apache.org list.
> >> Not sure why it didn't show up on dev - I sent msg to both lists.
> >>
> >> On Tue, Mar 15, 2022 at 3:41 PM Mike Beckerle 
> >> wrote:
> >>
> >> > No attached PDF.
> >> >
> >> > Removal of blanks from a number would normally be by way of DFDL
> >> "padding"
> >> > and "trimming" properties.
> >> >
> >> > You need dfdl:textNumberJustification property, also textTrimKind,
> >> > textPadKind, textNumberPadCharacter.
> >> >
> >> > You can also use textNumberPattern to indicate that a number may have
> >> > leading spaces or zeros, but this is mostly about output when those
> >> leading
> >> > zeros are required.
> >> >
> >> > Padding on left == Right Justified
> >> > Padding on right == Left Justified
> >> > Padding around both sides = Center Justified.
> >> >
> >> >
> >> >
> >> > On Tue, Mar 15, 2022 at 8:53 AM Attila Horvath <
> >> attila.j.horv...@gmail.com
> >> > >
> >> > wrote:
> >> >
> >> > > Ping... any assistance appreciated - thx
> >> > >
> >> > > -- Forwarded message -
> >> > > From: Attila Horvath 
> >> > > Date: Mon, Mar 14, 2022 at 12:38 PM
> >> > > Subject: string to integer conversion w/ leading blanks fails
> >> > > To: , 
> >> > >
> >> > >
> >> > > ALCON
> >> > >
> >> > > Can someone pls suggest a way to convert string to integer if/when
> >> > leading
> >> > > blanks are present?
> >> > >
> >> > > My attempts are failing - see attached pdf for more details.
> >> > >
> >> > > Thx in advance
> >> > >
> >> > > Attila
> >> > >
> >> >
> >>
> >
>

Re: string to integer conversion w/ leading blanks fails

[Attila Horvath]

Attachment can be found on
https://lists.apache.org/list.html?us...@daffodil.apache.org list.
Not sure why it didn't show up on dev - I sent msg to both lists.

On Tue, Mar 15, 2022 at 3:41 PM Mike Beckerle  wrote:

> No attached PDF.
>
> Removal of blanks from a number would normally be by way of DFDL "padding"
> and "trimming" properties.
>
> You need dfdl:textNumberJustification property, also textTrimKind,
> textPadKind, textNumberPadCharacter.
>
> You can also use textNumberPattern to indicate that a number may have
> leading spaces or zeros, but this is mostly about output when those leading
> zeros are required.
>
> Padding on left == Right Justified
> Padding on right == Left Justified
> Padding around both sides = Center Justified.
>
>
>
> On Tue, Mar 15, 2022 at 8:53 AM Attila Horvath  >
> wrote:
>
> > Ping... any assistance appreciated - thx
> >
> > -- Forwarded message -
> > From: Attila Horvath 
> > Date: Mon, Mar 14, 2022 at 12:38 PM
> > Subject: string to integer conversion w/ leading blanks fails
> > To: , 
> >
> >
> > ALCON
> >
> > Can someone pls suggest a way to convert string to integer if/when
> leading
> > blanks are present?
> >
> > My attempts are failing - see attached pdf for more details.
> >
> > Thx in advance
> >
> > Attila
> >
>

Fwd: string to integer conversion w/ leading blanks fails

[Attila Horvath]

Ping... any assistance appreciated - thx

-- Forwarded message -
From: Attila Horvath 
Date: Mon, Mar 14, 2022 at 12:38 PM
Subject: string to integer conversion w/ leading blanks fails
To: , 


ALCON

Can someone pls suggest a way to convert string to integer if/when leading
blanks are present?

My attempts are failing - see attached pdf for more details.

Thx in advance

Attila

string to integer conversion w/ leading blanks fails

[Attila Horvath]

ALCON

Can someone pls suggest a way to convert string to integer if/when leading
blanks are present?

My attempts are failing - see attached pdf for more details.

Thx in advance

Attila

Re: help processing missing optional data

[Attila Horvath]

PS:...

Case in point:...





When 'Optional line 3' is NOT there, I get an error on 2nd ''
because it resolves to an empty string.

<8(



On Tue, Mar 8, 2022 at 7:20 AM Attila Horvath 
wrote:

>
> ALCON
>
> I'm having difficulty parsing an input ASCII file containing optional
> records as follows:
>
>
>
>
>
>
>
> *Required line 1... fixed length Required line 2... fixed length
>  Optional line 3... fixed length  Optional comment... 1 of
> n  Optional comment... 2 of n  Optional comment... n of n
>  Required line 4... fixed length  :::*
>
> I've tried various  and lookahead approaches
> unsuccessfully and admittedly getting frustrated. <8(
>
> For example, I'm able to parse 'Optional line 3' *when it exists* BUT I
> can't construct DFDL script to process the file successfully when line 3 is
> not present.
>
> Can someone pls point me to an example/link illustrating best way to
> process this if/when optional lines are NOT present.
>
> Thx in advance
> Attila
>
>

help processing missing optional data

[Attila Horvath]

ALCON

I'm having difficulty parsing an input ASCII file containing optional
records as follows:







*Required line 1... fixed length Required line 2... fixed length
 Optional line 3... fixed length  Optional comment... 1 of
n  Optional comment... 2 of n  Optional comment... n of n
 Required line 4... fixed length  :::*

I've tried various  and lookahead approaches
unsuccessfully and admittedly getting frustrated. <8(

For example, I'm able to parse 'Optional line 3' *when it exists* BUT I
can't construct DFDL script to process the file successfully when line 3 is
not present.

Can someone pls point me to an example/link illustrating best way to
process this if/when optional lines are NOT present.

Thx in advance
Attila

Re: Fwd: FW: DFDL: potential problem

[Attila Horvath]

Given recent discussion re: releases

Any inclination to implement validation strategy for reconstituted data in a 
lossless environment?

Thx - Attila

On 2021/09/17 20:11:54, "Beckerle, Mike"  wrote: 
> Apologies on tardy reply. I missed parts of this thread due to spam email 
> filter.
> 
> (I learned that MS Outlook 365 is misclassifying some Apache email as junk 
> email. )
> 
> Here's the link to what is proposed for checksum calculations, and it has 
> links to some mock-ups showing how this checksum/crc stuff is supposed to 
> work.
> 
> https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Checksums%2C+CRC%2C+Parity+-+Layering+Enhancements
> 
> I do think this could be used to couple a generic hash into data that is 
> verified at unparse.
> 
> 
> 
> From: Steve Lawrence 
> Sent: Monday, August 30, 2021 9:50 AM
> To: dev@daffodil.apache.org 
> Subject: Re: Fwd: FW: DFDL: potential problem
> 
> Interesting idea.
> 
> I was thinking you could do something like this once we have this new
> feature implemented:
> 
>   
> 
>   
> 
>   
>dfdl:inputValueCalc="$checksum" />
>   
> 
>   http://www.ogf.org/dfdl/;>
> 
>   
> 
>   
> 
>   
> 
> So we parse and checksum the entire data foramt, add the checksum to the
> infoset via input value calc, and then add an assert that the calculated
> checksum matchs the value in the infoset.
> 
> On parse, these two should always be the same. But on unparse, it's
> possible they could be different and the assert would fail.
> Unfortunately, this doesn't actually work because assert's are evaluated
> during unparse.
> 
> This seems like a reasonable use case for asserts during unparse, and I
> imagine there are others, so maybe that's a feature worth considering to
> allow this type of unparse validation.
> 
> 
> 
> 
> On 8/25/21 9:20 AM, Attila Horvath wrote:
> >
> > *Subject:* DFDL: potential problem
> >
> > ALCON
> >
> > re: idea for checksum calculations in DFDL
> > <https://lists.apache.org/thread.html/r85112d45e552a1f5b467406a0f0a4bcaf143372b95c8e72f2669%40%3Cdev.daffodil.apache.org%3E>
> >
> > We may have a potential ‘situation’ as part of our DFDL/Daffodil offering as
> > follows…
> >
> > My DFDL schema development process consists of examining the exit codes of a
> > four (4) part mechanism:
> >
> >  1. DFDL parsing – “Houston, we have a go.”
> >  2. DFDL unparsing – “Houston, we have a go.”
> >  3. *End-to-end source/destination data comparison – “Houston, we have a 
> > problem.”*
> >  4. Intermediate xml validation against reconstituted data – “Houston, we 
> > have a
> > go.”
> >
> > I have an *_unintentional_*error in my DFDL schema- unfortunately the
> > data/schema is lost that created this situation. Per above, both parse and
> > unparse execute successfully and xmllint validates Daffodil’s intermediate 
> > XML
> > file successfully against the reconstituted/unparsed data as well against 
> > the
> > DFDL [erroneous] schema.
> >
> > However, the source and target data are *_NOT_* congruent.This is one 
> > situation
> > I did not anticipate this situation.
> >
> > This means, our model and incorporation of Daffodil in our situation leaves
> > [albeit] a /possibility/ to have an erroneous DFDL schema that will 
> > ultimately
> > send data end-to-end but because the two [gateway]ends do not
> > communicatedirectly w/ each other there is no way for the destination 
> > gateway to
> > verify if the data is identical w/ the data received by the source gateway.
> >
> > To address above and perhaps along the lines of 'checksum calculations' re: 
> > IPV4
> > element, what is the collective opinion of having a SHASUM capability added 
> > to
> > Daffodil allowing the parser to optionally ("invisibly") incorporate a 
> > SHASUM in
> > the intermediate XML file allowing the destination unparser to validate the
> > reconstitute the data against the incorporated SHASUM?
> >
> > Perhaps a lame suggestion, could Daffodil optionally insert a comment tag 
> > while
> > parsing identifying it as a Daffodil inserted shasum comment which the 
> > unparser
> > can identify and validate the reconstituted data.
> >
> > Thx in advance,
> >
> > v/r
> >
> > Attila
> >
> >
> 
> 

Fwd: FW: DFDL: potential problem

[Attila Horvath]

*Subject:* DFDL: potential problem

ALCON

re: idea for checksum calculations in DFDL


We may have a potential ‘situation’ as part of our DFDL/Daffodil offering
as follows…

My DFDL schema development process consists of examining the exit codes of
a four (4) part mechanism:


   1. DFDL parsing – “Houston, we have a go.”
   2. DFDL unparsing – “Houston, we have a go.”
   3. *End-to-end source/destination data comparison – “Houston, we have a
   problem.”*
   4. Intermediate xml validation against reconstituted data – “Houston, we
   have a go.”

I have an *unintentional* error in my DFDL schema - unfortunately the
data/schema is lost that created this situation. Per above, both parse and
unparse execute successfully and xmllint validates Daffodil’s intermediate
XML file successfully against the reconstituted/unparsed data as well
against the DFDL [erroneous] schema.

However, the source and target data are *NOT* congruent. This is one
situation I did not anticipate this situation.

This means, our model and incorporation of Daffodil in our situation
leaves [albeit]
a *possibility* to have an erroneous DFDL schema that will ultimately send
data end-to-end but because the two [gateway]  ends do not communicate
directly w/ each other there is no way for the destination gateway to
verify if the data is identical w/ the data received by the source gateway.

To address above and perhaps along the lines of 'checksum calculations' re:
IPV4 element, what is the collective opinion of having a SHASUM capability
added to Daffodil allowing the parser to optionally ("invisibly")
incorporate a SHASUM in the intermediate XML file allowing the destination
unparser to validate the reconstitute the data against the incorporated
SHASUM?

Perhaps a lame suggestion, could Daffodil optionally insert a comment tag
while parsing identifying it as a Daffodil inserted shasum comment which
the unparser can identify and validate the reconstituted data.

Thx in advance,

v/r

Attila

Re: daffodil issue reported

[Attila Horvath]

PS: log file has stack trace.

On Mon, Jul 26, 2021 at 11:18 AM Attila Horvath 
wrote:

> Steve
>
> Sorry - issue udpaed w/ new ~.zip to include defaults |AND| correct log
> file (had wrong log file before)
>
> Txh - Attila
>
>
> On Mon, Jul 26, 2021 at 11:01 AM Steve Lawrence 
> wrote:
>
>> Thanks for the report. I've commented on the issue, but just to make
>> sure you see it, could you provide the defaults.dfdl.xsd file and the
>> stack trace you get after the "An unexpected exception" error message?
>> That will help us reproduce and track down the issue.
>>
>> On 7/26/21 10:49 AM, Attila Horvath wrote:
>> > ALCON: pls see DAFFODIL-2549
>> > <https://issues.apache.org/jira/browse/DAFFODIL-2549>.
>> >
>> > Thx - Attila
>> >
>>
>>

Re: daffodil issue reported

[Attila Horvath]

Steve

Sorry - issue udpaed w/ new ~.zip to include defaults |AND| correct log
file (had wrong log file before)

Txh - Attila


On Mon, Jul 26, 2021 at 11:01 AM Steve Lawrence 
wrote:

> Thanks for the report. I've commented on the issue, but just to make
> sure you see it, could you provide the defaults.dfdl.xsd file and the
> stack trace you get after the "An unexpected exception" error message?
> That will help us reproduce and track down the issue.
>
> On 7/26/21 10:49 AM, Attila Horvath wrote:
> > ALCON: pls see DAFFODIL-2549
> > <https://issues.apache.org/jira/browse/DAFFODIL-2549>.
> >
> > Thx - Attila
> >
>
>

daffodil issue reported

[Attila Horvath]

ALCON: pls see DAFFODIL-2549
.

Thx - Attila

test cases - examples

[Attila Horvath]

ALCON

I assume there is a repository of test cases with which to validate and
regression test various corresponding daffodil features.

Are these test cases available to users for reference as examples?

Thx

Attila

Re: all this github spam ?

[Attila Horvath]

does github support (re)captcha on email submit?

On Wed, Apr 21, 2021 at 9:22 AM Steve Lawrence  wrote:

> Unfortunately, I'm not sure there's anything we can do about it.
>
> GitHub doesn't give any controls over who can/can't open a PR. We can't
> even temporarily close PR's completely.
>
> We could maybe make it so GitHub actions on PRs must be manually
> triggered so the spammers cryptocurrency mining stuff would never run.
> But that's a bit of a pain, and it relies on the spammers to realize
> their stuff isn't being run anymore and take us off their list. My guess
> is we're stuck on their list forever now.
>
> These crypto mining attacks are a known issue for GitHub, hopefully
> they're working on a solution. Tough, GitHub is eventually detecting
> these are spam and closing the accounts and deleting the PRS, but not
> until after the PR is created.
>
> As to the archive issue, we could maybe ask infra to remove archives
> that are clearly spam (all of them so far say "Demo titles Add
> files...", so unique and consistent). But it doesn't solve the
> underlying issue.
>
>
> On 4/21/21 8:59 AM, Beckerle, Mike wrote:
> > We seem to be fending off maybe 10 a day github spam attacks where
> people
> > open/close pull requests.
> >
> > Is there something systematic we can do to avoid this?
> >
> > This pollutes our mailing lists. I know we can manually purge the PRs
> from
> > github, but these things will live forever in the mail archives, adding
> a bunch
> > of random emails/account names to them, and generally making them less
> useful.
> >
> > Mike Beckerle | Principal Engineer
> >
> > mbecke...@owlcyberdefense.com 
> >
> > P +1-781-330-0412
> >
> > Connect with us!
> >
> > <
> https://twitter.com/owlcyberdefense>
> >
> > 
> >
> > **
> >
> > The information contained in this transmission is for the personal and
> > confidential use of the individual or entity to which it is addressed.
> If the
> > reader is not the intended recipient, you are hereby notified that any
> review,
> > dissemination, or copying of this communication is strictly prohibited.
> If you
> > have received this transmission in error, please notify the sender
> immediately
> >
>
>