AW: [OFFTOPIC] Re: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?

[Christofer Dutz]

Ahh ...

Think I'll renew my Office 365 subscripton and pass on that one ;-)

Duck and run ...

Chris

-Ursprüngliche Nachricht-
Von: Dave Fisher  
Gesendet: Donnerstag, 11. März 2021 21:55
An: dev@daffodil.apache.org
Betreff: [OFFTOPIC] Re: Acceptance criteria for merging DFDL-to-C backend 
(runtime2-2202)?

Hi Chris,

Try building OpenOffice … macOS, Windows, Linux,  and downstream FreeBSD, OS/2 
…. Older c++, two different build tools, patched dependencies.

;-)

Regards,
Dave

> On Mar 11, 2021, at 12:51 PM, Christofer Dutz  
> wrote:
> 
> Hi John,
> 
> the script is here:
> https://github.com/apache/plc4x/blob/develop/src/main/script/prerequisiteCheck.groovy
> 
> And it's called from here:
> https://github.com/apache/plc4x/blob/develop/pom.xml#L644
> 
> Hope it helps ... I just know as soon as you leave the Java eco-system 
> software-development really gets tricky regarding checking all the stuff that 
> could be or be configured wrong, so I tend to automate as much as possible.
> It increases the user acceptance and it reduces the questions from annoyed 
> users ;.)
> 
> Chris
> 
> 
> -Ursprüngliche Nachricht-
> Von: Interrante, John A (GE Research, US)  
> Gesendet: Donnerstag, 11. März 2021 20:51
> An: dev@daffodil.apache.org
> Betreff: RE: Acceptance criteria for merging DFDL-to-C backend 
> (runtime2-2202)?
> 
> Chris,
> 
> Yes, I think it would be nice if sbt called a Groovy or Scala script to check 
> that the C setup is correct.  Will you please tell me where to find PLC4X's 
> script and where it gets called from?
> 
> Thanks,
> John
> 
> -Original Message-
> From: Christofer Dutz  
> Sent: Thursday, March 11, 2021 11:54 AM
> To: dev@daffodil.apache.org
> Subject: EXT: AW: Acceptance criteria for merging DFDL-to-C backend 
> (runtime2-2202)?
> 
> Hi all,
> 
> I would strongly suggest having some tooling to check the setup is correct.
> In PLC4X we have a groovy script that is executed by the build. Depending on 
> which parts you are building, it checks if required tools are installed in 
> sufficient versions. Not sure if sbt allows similar checks, but perhaps our 
> setup in PLC4X could help?
> 
> Otherwise with all these C, C++ tools it's really hard for newbies to find 
> what's going wrong.
> 
> Chris
> 
> -Ursprüngliche Nachricht-
> Von: Beckerle, Mike  
> Gesendet: Donnerstag, 11. März 2021 17:38
> An: dev@daffodil.apache.org
> Betreff: Re: Acceptance criteria for merging DFDL-to-C backend 
> (runtime2-2202)?
> 
> Setup of the C-toolchain as well as scala shoulnd't be a problem so long as 
> these are all no-cost tools.
> 
> From: Interrante, John A (GE Research, US) 
> Sent: Thursday, March 11, 2021 11:34 AM
> To: dev@daffodil.apache.org 
> Subject: RE: Acceptance criteria for merging DFDL-to-C backend 
> (runtime2-2202)?
> 
> Is that all?  :-).  I would add some criteria testing runtime2's conformance 
> to the DFDL 1.0 specification as well.  Here goes...
> 
> 1) Sufficient functionality to describe at least one example, of sufficient 
> message complexity to indicate that other similar "real" examples should be 
> possible
> 
>Yup.  We have 7 schemas in 
> daffodil-test/src/test/resources/org/apache/daffodil/runtime2 that have 
> "real" examples of messages with sufficient complexity to indicate that other 
> real messages should be possible.
> 
> 2) contains built-in tests for one or several such examples, showing each 
> supported aspect working.
> 
>Yup.  We have TDML tests and binary data/XML infoset files for each of 
> these 7 schemas.
> 
> 3) the tests are fully integrated - they run every time 'sbt test' is run on 
> daffodil.
> 
>Yup.  We have Scala test classes for each of these 7 schemas' TDML 
> tests in daffodil-test/src/test/scala/org/apache/daffodil/runtime2.
> 
> 4) setup instructions for developers are there so that people know how to 
> insure the required C tool-chain elements are there. These need to work on 
> Linux and Windows.
> 
>Needs work.  The top-level README lists daffodil-runtime2's build 
> requirements under Build Requirements and has a single sentence under Getting 
> Started telling developers that they will need a C toolchain in order to 
> build daffodil-runtime2.  However, I didn't make it clear in the README that 
> developers must setup a C toolchain as well as a Java/Scala toolchain and I 
> need to add a "C Setup and Notes" page to the Confluence Wiki with 
> step-by-step instructions showing developers how to setup a C toolchain on 
> Linux and Windows.  I hope developers won't mind if we make it mandatory to 
> setup both Java/Scala and C toolchains before building Daffodil.  I don't 
> like the idea of some developers building Daffodil without checking that the 
> C files compile successfully and the runtime2 tests pass successfully.  I 
> think it's sufficient to guarantee that end-users of Daffodil never have to 
> install a C toolchain 

[OFFTOPIC] Re: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?

[Dave Fisher]

Hi Chris,

Try building OpenOffice … macOS, Windows, Linux,  and downstream FreeBSD, OS/2 
…. Older c++, two different build tools, patched dependencies.

;-)

Regards,
Dave

> On Mar 11, 2021, at 12:51 PM, Christofer Dutz  
> wrote:
> 
> Hi John,
> 
> the script is here:
> https://github.com/apache/plc4x/blob/develop/src/main/script/prerequisiteCheck.groovy
> 
> And it's called from here:
> https://github.com/apache/plc4x/blob/develop/pom.xml#L644
> 
> Hope it helps ... I just know as soon as you leave the Java eco-system 
> software-development really gets tricky regarding checking all the stuff that 
> could be or be configured wrong, so I tend to automate as much as possible.
> It increases the user acceptance and it reduces the questions from annoyed 
> users ;.)
> 
> Chris
> 
> 
> -Ursprüngliche Nachricht-
> Von: Interrante, John A (GE Research, US)  
> Gesendet: Donnerstag, 11. März 2021 20:51
> An: dev@daffodil.apache.org
> Betreff: RE: Acceptance criteria for merging DFDL-to-C backend 
> (runtime2-2202)?
> 
> Chris,
> 
> Yes, I think it would be nice if sbt called a Groovy or Scala script to check 
> that the C setup is correct.  Will you please tell me where to find PLC4X's 
> script and where it gets called from?
> 
> Thanks,
> John
> 
> -Original Message-
> From: Christofer Dutz  
> Sent: Thursday, March 11, 2021 11:54 AM
> To: dev@daffodil.apache.org
> Subject: EXT: AW: Acceptance criteria for merging DFDL-to-C backend 
> (runtime2-2202)?
> 
> Hi all,
> 
> I would strongly suggest having some tooling to check the setup is correct.
> In PLC4X we have a groovy script that is executed by the build. Depending on 
> which parts you are building, it checks if required tools are installed in 
> sufficient versions. Not sure if sbt allows similar checks, but perhaps our 
> setup in PLC4X could help?
> 
> Otherwise with all these C, C++ tools it's really hard for newbies to find 
> what's going wrong.
> 
> Chris
> 
> -Ursprüngliche Nachricht-
> Von: Beckerle, Mike  
> Gesendet: Donnerstag, 11. März 2021 17:38
> An: dev@daffodil.apache.org
> Betreff: Re: Acceptance criteria for merging DFDL-to-C backend 
> (runtime2-2202)?
> 
> Setup of the C-toolchain as well as scala shoulnd't be a problem so long as 
> these are all no-cost tools.
> 
> From: Interrante, John A (GE Research, US) 
> Sent: Thursday, March 11, 2021 11:34 AM
> To: dev@daffodil.apache.org 
> Subject: RE: Acceptance criteria for merging DFDL-to-C backend 
> (runtime2-2202)?
> 
> Is that all?  :-).  I would add some criteria testing runtime2's conformance 
> to the DFDL 1.0 specification as well.  Here goes...
> 
> 1) Sufficient functionality to describe at least one example, of sufficient 
> message complexity to indicate that other similar "real" examples should be 
> possible
> 
>Yup.  We have 7 schemas in 
> daffodil-test/src/test/resources/org/apache/daffodil/runtime2 that have 
> "real" examples of messages with sufficient complexity to indicate that other 
> real messages should be possible.
> 
> 2) contains built-in tests for one or several such examples, showing each 
> supported aspect working.
> 
>Yup.  We have TDML tests and binary data/XML infoset files for each of 
> these 7 schemas.
> 
> 3) the tests are fully integrated - they run every time 'sbt test' is run on 
> daffodil.
> 
>Yup.  We have Scala test classes for each of these 7 schemas' TDML 
> tests in daffodil-test/src/test/scala/org/apache/daffodil/runtime2.
> 
> 4) setup instructions for developers are there so that people know how to 
> insure the required C tool-chain elements are there. These need to work on 
> Linux and Windows.
> 
>Needs work.  The top-level README lists daffodil-runtime2's build 
> requirements under Build Requirements and has a single sentence under Getting 
> Started telling developers that they will need a C toolchain in order to 
> build daffodil-runtime2.  However, I didn't make it clear in the README that 
> developers must setup a C toolchain as well as a Java/Scala toolchain and I 
> need to add a "C Setup and Notes" page to the Confluence Wiki with 
> step-by-step instructions showing developers how to setup a C toolchain on 
> Linux and Windows.  I hope developers won't mind if we make it mandatory to 
> setup both Java/Scala and C toolchains before building Daffodil.  I don't 
> like the idea of some developers building Daffodil without checking that the 
> C files compile successfully and the runtime2 tests pass successfully.  I 
> think it's sufficient to guarantee that end-users of Daffodil never have to 
> install a C toolchain to use Daffodil.  When end-users install Daffodil, they 
> will get a pure Scala Daffodil just like before.  The only user-visible 
> change is that this Daffodil will know how to generate, compile, and run C 
> code on the end-user's system if the user asks Daffodil to do that.
> 
> I also would add another criteria:
> 

AW: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?

[Christofer Dutz]

Hi John,

the script is here:
https://github.com/apache/plc4x/blob/develop/src/main/script/prerequisiteCheck.groovy

And it's called from here:
https://github.com/apache/plc4x/blob/develop/pom.xml#L644

Hope it helps ... I just know as soon as you leave the Java eco-system 
software-development really gets tricky regarding checking all the stuff that 
could be or be configured wrong, so I tend to automate as much as possible.
It increases the user acceptance and it reduces the questions from annoyed 
users ;.)

Chris


-Ursprüngliche Nachricht-
Von: Interrante, John A (GE Research, US)  
Gesendet: Donnerstag, 11. März 2021 20:51
An: dev@daffodil.apache.org
Betreff: RE: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?

Chris,

Yes, I think it would be nice if sbt called a Groovy or Scala script to check 
that the C setup is correct.  Will you please tell me where to find PLC4X's 
script and where it gets called from?

Thanks,
John

-Original Message-
From: Christofer Dutz  
Sent: Thursday, March 11, 2021 11:54 AM
To: dev@daffodil.apache.org
Subject: EXT: AW: Acceptance criteria for merging DFDL-to-C backend 
(runtime2-2202)?

Hi all,

I would strongly suggest having some tooling to check the setup is correct.
In PLC4X we have a groovy script that is executed by the build. Depending on 
which parts you are building, it checks if required tools are installed in 
sufficient versions. Not sure if sbt allows similar checks, but perhaps our 
setup in PLC4X could help?

Otherwise with all these C, C++ tools it's really hard for newbies to find 
what's going wrong.

Chris

-Ursprüngliche Nachricht-
Von: Beckerle, Mike  
Gesendet: Donnerstag, 11. März 2021 17:38
An: dev@daffodil.apache.org
Betreff: Re: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?

Setup of the C-toolchain as well as scala shoulnd't be a problem so long as 
these are all no-cost tools.

From: Interrante, John A (GE Research, US) 
Sent: Thursday, March 11, 2021 11:34 AM
To: dev@daffodil.apache.org 
Subject: RE: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?

Is that all?  :-).  I would add some criteria testing runtime2's conformance to 
the DFDL 1.0 specification as well.  Here goes...

1) Sufficient functionality to describe at least one example, of sufficient 
message complexity to indicate that other similar "real" examples should be 
possible

Yup.  We have 7 schemas in 
daffodil-test/src/test/resources/org/apache/daffodil/runtime2 that have "real" 
examples of messages with sufficient complexity to indicate that other real 
messages should be possible.

2) contains built-in tests for one or several such examples, showing each 
supported aspect working.

Yup.  We have TDML tests and binary data/XML infoset files for each of 
these 7 schemas.

3) the tests are fully integrated - they run every time 'sbt test' is run on 
daffodil.

Yup.  We have Scala test classes for each of these 7 schemas' TDML 
tests in daffodil-test/src/test/scala/org/apache/daffodil/runtime2.

4) setup instructions for developers are there so that people know how to 
insure the required C tool-chain elements are there. These need to work on 
Linux and Windows.

Needs work.  The top-level README lists daffodil-runtime2's build 
requirements under Build Requirements and has a single sentence under Getting 
Started telling developers that they will need a C toolchain in order to build 
daffodil-runtime2.  However, I didn't make it clear in the README that 
developers must setup a C toolchain as well as a Java/Scala toolchain and I 
need to add a "C Setup and Notes" page to the Confluence Wiki with step-by-step 
instructions showing developers how to setup a C toolchain on Linux and 
Windows.  I hope developers won't mind if we make it mandatory to setup both 
Java/Scala and C toolchains before building Daffodil.  I don't like the idea of 
some developers building Daffodil without checking that the C files compile 
successfully and the runtime2 tests pass successfully.  I think it's sufficient 
to guarantee that end-users of Daffodil never have to install a C toolchain to 
use Daffodil.  When end-users install Daffodil, they will get a pure Scala 
Daffodil just like before.  The only user-visible change is that this Daffodil 
will know how to generate, compile, and run C code on the end-user's system if 
the user asks Daffodil to do that.

I also would add another criteria:

5) A subset of Daffodil's DFDL 1.0 specification conformance TDML tests 
modified to run on runtime2 as well as runtime1 during 'sbt test'.

We have lots and lots of TDML tests checking that Daffodil conforms to the DFDL 
1.0 specification.  We should be able to find a subset of these tests that 
should pass on runtime2 as well and verify that they do keep passing when run 
on both runtime1 and runtime2.  This step would go a long way to ensure that 
people 

RE: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?

[Interrante, John A (GE Research, US)]

Chris,

Yes, I think it would be nice if sbt called a Groovy or Scala script to check 
that the C setup is correct.  Will you please tell me where to find PLC4X's 
script and where it gets called from?

Thanks,
John

-Original Message-
From: Christofer Dutz  
Sent: Thursday, March 11, 2021 11:54 AM
To: dev@daffodil.apache.org
Subject: EXT: AW: Acceptance criteria for merging DFDL-to-C backend 
(runtime2-2202)?

Hi all,

I would strongly suggest having some tooling to check the setup is correct.
In PLC4X we have a groovy script that is executed by the build. Depending on 
which parts you are building, it checks if required tools are installed in 
sufficient versions. Not sure if sbt allows similar checks, but perhaps our 
setup in PLC4X could help?

Otherwise with all these C, C++ tools it's really hard for newbies to find 
what's going wrong.

Chris

-Ursprüngliche Nachricht-
Von: Beckerle, Mike  
Gesendet: Donnerstag, 11. März 2021 17:38
An: dev@daffodil.apache.org
Betreff: Re: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?

Setup of the C-toolchain as well as scala shoulnd't be a problem so long as 
these are all no-cost tools.

From: Interrante, John A (GE Research, US) 
Sent: Thursday, March 11, 2021 11:34 AM
To: dev@daffodil.apache.org 
Subject: RE: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?

Is that all?  :-).  I would add some criteria testing runtime2's conformance to 
the DFDL 1.0 specification as well.  Here goes...

1) Sufficient functionality to describe at least one example, of sufficient 
message complexity to indicate that other similar "real" examples should be 
possible

Yup.  We have 7 schemas in 
daffodil-test/src/test/resources/org/apache/daffodil/runtime2 that have "real" 
examples of messages with sufficient complexity to indicate that other real 
messages should be possible.

2) contains built-in tests for one or several such examples, showing each 
supported aspect working.

Yup.  We have TDML tests and binary data/XML infoset files for each of 
these 7 schemas.

3) the tests are fully integrated - they run every time 'sbt test' is run on 
daffodil.

Yup.  We have Scala test classes for each of these 7 schemas' TDML 
tests in daffodil-test/src/test/scala/org/apache/daffodil/runtime2.

4) setup instructions for developers are there so that people know how to 
insure the required C tool-chain elements are there. These need to work on 
Linux and Windows.

Needs work.  The top-level README lists daffodil-runtime2's build 
requirements under Build Requirements and has a single sentence under Getting 
Started telling developers that they will need a C toolchain in order to build 
daffodil-runtime2.  However, I didn't make it clear in the README that 
developers must setup a C toolchain as well as a Java/Scala toolchain and I 
need to add a "C Setup and Notes" page to the Confluence Wiki with step-by-step 
instructions showing developers how to setup a C toolchain on Linux and 
Windows.  I hope developers won't mind if we make it mandatory to setup both 
Java/Scala and C toolchains before building Daffodil.  I don't like the idea of 
some developers building Daffodil without checking that the C files compile 
successfully and the runtime2 tests pass successfully.  I think it's sufficient 
to guarantee that end-users of Daffodil never have to install a C toolchain to 
use Daffodil.  When end-users install Daffodil, they will get a pure Scala 
Daffodil just like before.  The only user-visible change is that this Daffodil 
will know how to generate, compile, and run C code on the end-user's system if 
the user asks Daffodil to do that.

I also would add another criteria:

5) A subset of Daffodil's DFDL 1.0 specification conformance TDML tests 
modified to run on runtime2 as well as runtime1 during 'sbt test'.

We have lots and lots of TDML tests checking that Daffodil conforms to the DFDL 
1.0 specification.  We should be able to find a subset of these tests that 
should pass on runtime2 as well and verify that they do keep passing when run 
on both runtime1 and runtime2.  This step would go a long way to ensure that 
people working on other aspects of Daffodil do not break the C backend 
inadvertently without knowing it.  Note that this makes it even more mandatory 
that Daffodil developers setup both Java/Scala and C toolchains.  What do you 
all think?


-Original Message-
From: Beckerle, Mike 
Sent: Thursday, March 11, 2021 9:27 AM
To: dev@daffodil.apache.org
Subject: EXT: Re: Acceptance criteria for merging DFDL-to-C backend 
(runtime2-2202)?

Acceptance criteria to me:

1) Sufficient functionality to describe at least one example, of sufficient 
message complexity to indicate that other similar "real" examples should be 
possible

2) contains built-in tests for one or several such examples, showing each 
supported aspect working.

3) the tests are fully 

AW: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?

[Christofer Dutz]

Hi all,

I would strongly suggest having some tooling to check the setup is correct.
In PLC4X we have a groovy script that is executed by the build. Depending on 
which parts you are building, it checks if required tools are installed in 
sufficient versions. Not sure if sbt allows similar checks, but perhaps our 
setup in PLC4X could help?

Otherwise with all these C, C++ tools it's really hard for newbies to find 
what's going wrong.

Chris

-Ursprüngliche Nachricht-
Von: Beckerle, Mike  
Gesendet: Donnerstag, 11. März 2021 17:38
An: dev@daffodil.apache.org
Betreff: Re: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?

Setup of the C-toolchain as well as scala shoulnd't be a problem so long as 
these are all no-cost tools.

From: Interrante, John A (GE Research, US) 
Sent: Thursday, March 11, 2021 11:34 AM
To: dev@daffodil.apache.org 
Subject: RE: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?

Is that all?  :-).  I would add some criteria testing runtime2's conformance to 
the DFDL 1.0 specification as well.  Here goes...

1) Sufficient functionality to describe at least one example, of sufficient 
message complexity to indicate that other similar "real" examples should be 
possible

Yup.  We have 7 schemas in 
daffodil-test/src/test/resources/org/apache/daffodil/runtime2 that have "real" 
examples of messages with sufficient complexity to indicate that other real 
messages should be possible.

2) contains built-in tests for one or several such examples, showing each 
supported aspect working.

Yup.  We have TDML tests and binary data/XML infoset files for each of 
these 7 schemas.

3) the tests are fully integrated - they run every time 'sbt test' is run on 
daffodil.

Yup.  We have Scala test classes for each of these 7 schemas' TDML 
tests in daffodil-test/src/test/scala/org/apache/daffodil/runtime2.

4) setup instructions for developers are there so that people know how to 
insure the required C tool-chain elements are there. These need to work on 
Linux and Windows.

Needs work.  The top-level README lists daffodil-runtime2's build 
requirements under Build Requirements and has a single sentence under Getting 
Started telling developers that they will need a C toolchain in order to build 
daffodil-runtime2.  However, I didn't make it clear in the README that 
developers must setup a C toolchain as well as a Java/Scala toolchain and I 
need to add a "C Setup and Notes" page to the Confluence Wiki with step-by-step 
instructions showing developers how to setup a C toolchain on Linux and 
Windows.  I hope developers won't mind if we make it mandatory to setup both 
Java/Scala and C toolchains before building Daffodil.  I don't like the idea of 
some developers building Daffodil without checking that the C files compile 
successfully and the runtime2 tests pass successfully.  I think it's sufficient 
to guarantee that end-users of Daffodil never have to install a C toolchain to 
use Daffodil.  When end-users install Daffodil, they will get a pure Scala 
Daffodil just like before.  The only user-visible change is that this Daffodil 
will know how to generate, compile, and run C code on the end-user's system if 
the user asks Daffodil to do that.

I also would add another criteria:

5) A subset of Daffodil's DFDL 1.0 specification conformance TDML tests 
modified to run on runtime2 as well as runtime1 during 'sbt test'.

We have lots and lots of TDML tests checking that Daffodil conforms to the DFDL 
1.0 specification.  We should be able to find a subset of these tests that 
should pass on runtime2 as well and verify that they do keep passing when run 
on both runtime1 and runtime2.  This step would go a long way to ensure that 
people working on other aspects of Daffodil do not break the C backend 
inadvertently without knowing it.  Note that this makes it even more mandatory 
that Daffodil developers setup both Java/Scala and C toolchains.  What do you 
all think?


-Original Message-
From: Beckerle, Mike 
Sent: Thursday, March 11, 2021 9:27 AM
To: dev@daffodil.apache.org
Subject: EXT: Re: Acceptance criteria for merging DFDL-to-C backend 
(runtime2-2202)?

Acceptance criteria to me:

1) Sufficient functionality to describe at least one example, of sufficient 
message complexity to indicate that other similar "real" examples should be 
possible

2) contains built-in tests for one or several such examples, showing each 
supported aspect working.

3) the tests are fully integrated - they run every time 'sbt test' is run on 
daffodil.

4) setup instructions for developers are there so that people know how to 
insure the required C tool-chain elements are there. These need to work on 
Linux and Windows.

I believe these things insure that people working on other aspects of Daffodil 
will not be inadvertently breaking the C backend indetectably. That's one major 
concern I would have.

Re: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?

[Beckerle, Mike]

Setup of the C-toolchain as well as scala shoulnd't be a problem so long as 
these are all no-cost tools.

From: Interrante, John A (GE Research, US) 
Sent: Thursday, March 11, 2021 11:34 AM
To: dev@daffodil.apache.org 
Subject: RE: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?

Is that all?  :-).  I would add some criteria testing runtime2's conformance to 
the DFDL 1.0 specification as well.  Here goes...

1) Sufficient functionality to describe at least one example, of sufficient 
message complexity to indicate that other similar "real" examples should be 
possible

Yup.  We have 7 schemas in 
daffodil-test/src/test/resources/org/apache/daffodil/runtime2 that have "real" 
examples of messages with sufficient complexity to indicate that other real 
messages should be possible.

2) contains built-in tests for one or several such examples, showing each 
supported aspect working.

Yup.  We have TDML tests and binary data/XML infoset files for each of 
these 7 schemas.

3) the tests are fully integrated - they run every time 'sbt test' is run on 
daffodil.

Yup.  We have Scala test classes for each of these 7 schemas' TDML 
tests in daffodil-test/src/test/scala/org/apache/daffodil/runtime2.

4) setup instructions for developers are there so that people know how to 
insure the required C tool-chain elements are there. These need to work on 
Linux and Windows.

Needs work.  The top-level README lists daffodil-runtime2's build 
requirements under Build Requirements and has a single sentence under Getting 
Started telling developers that they will need a C toolchain in order to build 
daffodil-runtime2.  However, I didn't make it clear in the README that 
developers must setup a C toolchain as well as a Java/Scala toolchain and I 
need to add a "C Setup and Notes" page to the Confluence Wiki with step-by-step 
instructions showing developers how to setup a C toolchain on Linux and 
Windows.  I hope developers won't mind if we make it mandatory to setup both 
Java/Scala and C toolchains before building Daffodil.  I don't like the idea of 
some developers building Daffodil without checking that the C files compile 
successfully and the runtime2 tests pass successfully.  I think it's sufficient 
to guarantee that end-users of Daffodil never have to install a C toolchain to 
use Daffodil.  When end-users install Daffodil, they will get a pure Scala 
Daffodil just like before.  The only user-visible change is that this Daffodil 
will know how to generate, compile, and run C code on the end-user's system if 
the user asks Daffodil to do that.

I also would add another criteria:

5) A subset of Daffodil's DFDL 1.0 specification conformance TDML tests 
modified to run on runtime2 as well as runtime1 during 'sbt test'.

We have lots and lots of TDML tests checking that Daffodil conforms to the DFDL 
1.0 specification.  We should be able to find a subset of these tests that 
should pass on runtime2 as well and verify that they do keep passing when run 
on both runtime1 and runtime2.  This step would go a long way to ensure that 
people working on other aspects of Daffodil do not break the C backend 
inadvertently without knowing it.  Note that this makes it even more mandatory 
that Daffodil developers setup both Java/Scala and C toolchains.  What do you 
all think?


-Original Message-
From: Beckerle, Mike 
Sent: Thursday, March 11, 2021 9:27 AM
To: dev@daffodil.apache.org
Subject: EXT: Re: Acceptance criteria for merging DFDL-to-C backend 
(runtime2-2202)?

Acceptance criteria to me:

1) Sufficient functionality to describe at least one example, of sufficient 
message complexity to indicate that other similar "real" examples should be 
possible

2) contains built-in tests for one or several such examples, showing each 
supported aspect working.

3) the tests are fully integrated - they run every time 'sbt test' is run on 
daffodil.

4) setup instructions for developers are there so that people know how to 
insure the required C tool-chain elements are there. These need to work on 
Linux and Windows.

I believe these things insure that people working on other aspects of Daffodil 
will not be inadvertently breaking the C backend indetectably. That's one major 
concern I would have.

From: Interrante, John A (GE Research, US) 
Sent: Wednesday, March 10, 2021 4:24 PM
To: dev@daffodil.apache.org 
Subject: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?

Thanks to Mike's suggestion, I have moved the Runtime2 ToDos (changes requested 
by reviewers) from the 
DAFFODIL-2202 issue to an 
AsciiDoc document in the dev/design-notes subtree of the Daffodil website which 
you can read here:

https://daffodil.apache.org/dev/design-notes/runtime2-todos/

I would like to discuss which acceptance criteria the runtime2-2202 development 
branch 

RE: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?

[Interrante, John A (GE Research, US)]

Is that all?  :-).  I would add some criteria testing runtime2's conformance to 
the DFDL 1.0 specification as well.  Here goes...

1) Sufficient functionality to describe at least one example, of sufficient 
message complexity to indicate that other similar "real" examples should be 
possible

Yup.  We have 7 schemas in 
daffodil-test/src/test/resources/org/apache/daffodil/runtime2 that have "real" 
examples of messages with sufficient complexity to indicate that other real 
messages should be possible.

2) contains built-in tests for one or several such examples, showing each 
supported aspect working.

Yup.  We have TDML tests and binary data/XML infoset files for each of 
these 7 schemas.

3) the tests are fully integrated - they run every time 'sbt test' is run on 
daffodil.

Yup.  We have Scala test classes for each of these 7 schemas' TDML 
tests in daffodil-test/src/test/scala/org/apache/daffodil/runtime2.

4) setup instructions for developers are there so that people know how to 
insure the required C tool-chain elements are there. These need to work on 
Linux and Windows.

Needs work.  The top-level README lists daffodil-runtime2's build 
requirements under Build Requirements and has a single sentence under Getting 
Started telling developers that they will need a C toolchain in order to build 
daffodil-runtime2.  However, I didn't make it clear in the README that 
developers must setup a C toolchain as well as a Java/Scala toolchain and I 
need to add a "C Setup and Notes" page to the Confluence Wiki with step-by-step 
instructions showing developers how to setup a C toolchain on Linux and 
Windows.  I hope developers won't mind if we make it mandatory to setup both 
Java/Scala and C toolchains before building Daffodil.  I don't like the idea of 
some developers building Daffodil without checking that the C files compile 
successfully and the runtime2 tests pass successfully.  I think it's sufficient 
to guarantee that end-users of Daffodil never have to install a C toolchain to 
use Daffodil.  When end-users install Daffodil, they will get a pure Scala 
Daffodil just like before.  The only user-visible change is that this Daffodil 
will know how to generate, compile, and run C code on the end-user's system if 
the user asks Daffodil to do that.

I also would add another criteria:

5) A subset of Daffodil's DFDL 1.0 specification conformance TDML tests 
modified to run on runtime2 as well as runtime1 during 'sbt test'.

We have lots and lots of TDML tests checking that Daffodil conforms to the DFDL 
1.0 specification.  We should be able to find a subset of these tests that 
should pass on runtime2 as well and verify that they do keep passing when run 
on both runtime1 and runtime2.  This step would go a long way to ensure that 
people working on other aspects of Daffodil do not break the C backend 
inadvertently without knowing it.  Note that this makes it even more mandatory 
that Daffodil developers setup both Java/Scala and C toolchains.  What do you 
all think?


-Original Message-
From: Beckerle, Mike  
Sent: Thursday, March 11, 2021 9:27 AM
To: dev@daffodil.apache.org
Subject: EXT: Re: Acceptance criteria for merging DFDL-to-C backend 
(runtime2-2202)?

Acceptance criteria to me:

1) Sufficient functionality to describe at least one example, of sufficient 
message complexity to indicate that other similar "real" examples should be 
possible

2) contains built-in tests for one or several such examples, showing each 
supported aspect working.

3) the tests are fully integrated - they run every time 'sbt test' is run on 
daffodil.

4) setup instructions for developers are there so that people know how to 
insure the required C tool-chain elements are there. These need to work on 
Linux and Windows.

I believe these things insure that people working on other aspects of Daffodil 
will not be inadvertently breaking the C backend indetectably. That's one major 
concern I would have.

From: Interrante, John A (GE Research, US) 
Sent: Wednesday, March 10, 2021 4:24 PM
To: dev@daffodil.apache.org 
Subject: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?

Thanks to Mike's suggestion, I have moved the Runtime2 ToDos (changes requested 
by reviewers) from the 
DAFFODIL-2202 issue to an 
AsciiDoc document in the dev/design-notes subtree of the Daffodil website which 
you can read here:

https://daffodil.apache.org/dev/design-notes/runtime2-todos/

I would like to discuss which acceptance criteria the runtime2-2202 development 
branch must meet before I can submit a pull request to merge the DFDL-to-C 
backend and code generator into the main branch.  I plan to address the 
Runtime2 ToDos and I especially want to run some of Daffodil's TDML tests on 
the new runtime2 backend as well as the runtime1 backend by adding 
defaultImplementations="daffodil 

Re: Request JIRA access

[Steve Lawrence]

Welcome! You've been added.

I also suggest you subscribe to the dev list by sending an email to:

  dev-subscr...@daffodil.apache.org

This is the primary place where most development discussions take place.
The other place being JIRA tickets/Github pull requests that show up on
comm...@daffodil.apache.org. So it may also be worth subscribing to that
by sending an email to

  commits-subscr...@daffodil.apache.org

- Steve

On 3/11/21 10:37 AM, Mike Hoke wrote:
> Hi,
> 
> I would like to contribute to the Daffodil project! Please grant me Jira 
> contributor access.
> 
> My Jira user name is: mhoke
> 
> Thanks,
> Mike
> 

Request JIRA access

[Mike Hoke]

Hi,

I would like to contribute to the Daffodil project! Please grant me Jira 
contributor access.

My Jira user name is: mhoke

Thanks,
Mike

Re: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?

[Beckerle, Mike]

Acceptance criteria to me:

1) Sufficient functionality to describe at least one example, of sufficient 
message complexity to indicate that other similar "real" examples should be 
possible

2) contains built-in tests for one or several such examples, showing each 
supported aspect working.

3) the tests are fully integrated - they run every time 'sbt test' is run on 
daffodil.

4) setup instructions for developers are there so that people know how to 
insure the required C tool-chain elements are there. These need to work on 
Linux and Windows.

I believe these things insure that people working on other aspects of Daffodil 
will not be inadvertently breaking the C backend indetectably. That's one major 
concern I would have.

From: Interrante, John A (GE Research, US) 
Sent: Wednesday, March 10, 2021 4:24 PM
To: dev@daffodil.apache.org 
Subject: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?

Thanks to Mike's suggestion, I have moved the Runtime2 ToDos (changes requested 
by reviewers) from the 
DAFFODIL-2202 issue to an 
AsciiDoc document in the dev/design-notes subtree of the Daffodil website which 
you can read here:

https://daffodil.apache.org/dev/design-notes/runtime2-todos/

I would like to discuss which acceptance criteria the runtime2-2202 development 
branch must meet before I can submit a pull request to merge the DFDL-to-C 
backend and code generator into the main branch.  I plan to address the 
Runtime2 ToDos and I especially want to run some of Daffodil's TDML tests on 
the new runtime2 backend as well as the runtime1 backend by adding 
defaultImplementations="daffodil daffodil-runtime2" to certain TDML tests' 
attributes.  (Although I suggest we use the shorter name "daf-c" or "runtime2" 
because "daffodil daffodil-runtime2" is a lot of characters to put into the 
defaultImplementations attribute.)

Daffodil's Confluence describes Runtime2's design here:

https://cwiki.apache.org/confluence/display/DAFFODIL/WIP:+Daffodil+Runtime+2

In particular, it suggests we divide the implementation of runtime2 into two 
distinct phases:

  *   Phase 1 (aka Runtime2P1): No expressions. All lengths are fixed. All 
arrays have fixed length.
  *   Phase 2 (aka Runtime2P2): Adding the DFDL expression language, lengthKind 
'explicit', occursCountKind 'expression'.
I think phase 1 is almost done but we need to run a subset of Daffodil's TDML 
tests on runtime2 before we can really say for sure.
Here is an initial set of discussion points - more questions and criteria are 
welcome:


  1.  Which Daffodil TDML tests do we need to run on runtime2 to assert that 
phase 1 is complete?
  2.  Can we merge runtime2 when these tests pass and then build out phase 2 in 
the main branch, hopefully with help from other developers once they see how 
useful phase 1 is?
  3.  Which of the Runtime2 ToDos need to be done before the merge as well?

Once we agree on a minimal set of acceptance criteria for the merge, I'll copy 
the criteria to the JIRA issue.