Re: [Discuss] creating Release 3.1.0 and 96 JIRA tickets marked "Major" or higher

2021-04-20 Thread Beckerle, Mike 
Besides those 2 bugs, I think we should also merge 
https://github.com/apache/daffodil/pull/490
which adds a new charset (EBCDIC). We should roll this forward if susmita can't 
take it up.

From: Beckerle, Mike 
Sent: Tuesday, April 20, 2021 3:49 PM
To: dev@daffodil.apache.org 
Subject: Re: [Discuss] creating Release 3.1.0 and 96 JIRA tickets marked 
"Major" or higher


Looking at the critical JIRA tickets, they are:

DAFFODIL-1422 disallow 
doctype decls in all XML & XSD that we read in
DAFFODIL-2473 Need 
bit-wise AND, OR, NOT, and shift operations
DAFFODIL-2183 Unparse 
nilled complex element fails.
DAFFODIL-1598 Unparser: 
For strings that truncate, the dfdl:valueLength function cannot suspend
DAFFODIL-2399 Error 
diagnostics output even though there is an infoset
DAFFODIL-2400 New SAX API 
causes performance degredations.
DAFFODIL-1971 Statement 
order of evaluation not per DFDL Spec

I'm going to suggest how we handle each of these for the 3.1.0 release.

I end up with 2 bugs to fix.

Keep in mind this is discussion fodder.

DAFFODIL-1422 disallow 
doctype decls in all XML & XSD that we read in

  *   release note it - try to fix for 3.2.0

DAFFODIL-2473 Need 
bit-wise AND, OR, NOT, and shift operations

  *   change to priority major - we need better documentation of the use case 
driving this.

DAFFODIL-2183 Unparse 
nilled complex element fails.

  *   Fix. This is actually a pretty bad bug, and some solutions to problems 
that have come up in real schemas need to use nilled complex elements in the 
solution.

DAFFODIL-1598 Unparser: 
For strings that truncate, the dfdl:valueLength function cannot suspend

  *Release Note, leave open. I question if this is critical. Change 
priority?

DAFFODIL-2399 Error 
diagnostics output even though there is an infoset

  *Fix. This is actually a pretty bad bug.

DAFFODIL-2400 New SAX API 
causes performance degredations.

  *   Release Note. Change priority to Major. I think this should not be 
critical as SAX is a relatively new feature, and while SAX is a 
performance-oriented feature, it's not like this is a regression on current 
functionality.

DAFFODIL-1971 Statement 
order of evaluation not per DFDL Spec

  *   Release Note.

Comments?


From: Interrante, John A (GE Research, US) 
Sent: Monday, April 12, 2021 2:52 PM
To: dev@daffodil.apache.org 
Subject: [Discuss] creating Release 3.1.0 and 96 JIRA tickets marked "Major" or 
higher

I found the list of open and in-progress issues sorted by priority 
(https://issues.apache.org/jira/issues/?jql=project%20%3D%20DAFFODIL%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC).
  I looked at all the critical issues (there are 8 in this list) and I don't 
think we have to hold up 3.1.0 for them although it would be nice if some got 
fixed first (e.g., DAFFODIL-1422 disallowing doctype decls for security and 
DAFFODIL-2400 fixing SAX API conformance and performance).

However, I would like to merge the new Runtime2 backend in time for the 3.1.0 
release of Daffodil in order to 1) accelerate development by avoiding the need 
to rebase the runtime2 branch on the master branch periodically, and 2) attract 
new developers to help me build out the runtime2 code further.  I need to 
finish a review of my outstanding pull request, merge it, rebase runtime2 on 
the master branch, submit a pull request to merge runtime2, and complete that 
PR’s review, so about 1-2 more weeks.

John

-Original Message-
From: Steve Lawrence 
Sent: Monday, April 12, 2021 1:33 PM
To: dev@daffodil.apache.org
Subject: EXT: Re: [Discuss] creating Release 3.1.0 and 96 JIRA tickets marked 
"Major" or higher

I would also like to see the updates to SAX merged--there's some important API 
conformance and performance fixes that are worth getting in. There's an open 
pull request (PR #478) that I think is close to being ready to be merged.

Of the open critical issues, I think DAFFODIL-2399 might be worth blocking if 
we were close to a fix. But I don't think anyone is actively working on it and 
I don't believe we ever tracked down what

Give a virtual talk at APACHECON about Daffodil?

2021-04-20 Thread Beckerle, Mike 
ApacheCon @ Home 2021 is Sept 21-23.  
https://www.apachecon.com/acah2021/index.html

Consider giving a presentation related to Apache Daffodil ! The call for 
presentations is open until May 
2nd Midnight EDT.US (= 05:00 UTC). You need only formulate the idea for your 
presentation and submit a proposal for your presentation by that time.

This is a virtual conference, so you will be giving the presentation over an 
online system that records it. Makes a youtube video of it, etc.

If you have ideas for a presentation that you'd like to collaborate on, let's 
discuss!


Mike Beckerle | Principal Engineer

[cid:4850e7cc-d259-4fe0-bcaa-53a5dcee8e4b]

mbecke...@owlcyberdefense.com

P +1-781-330-0412

Re: [Discuss] creating Release 3.1.0 and 96 JIRA tickets marked "Major" or higher

2021-04-20 Thread Beckerle, Mike 

Looking at the critical JIRA tickets, they are:

DAFFODIL-1422 disallow 
doctype decls in all XML & XSD that we read in
DAFFODIL-2473 Need 
bit-wise AND, OR, NOT, and shift operations
DAFFODIL-2183 Unparse 
nilled complex element fails.
DAFFODIL-1598 Unparser: 
For strings that truncate, the dfdl:valueLength function cannot suspend
DAFFODIL-2399 Error 
diagnostics output even though there is an infoset
DAFFODIL-2400 New SAX API 
causes performance degredations.
DAFFODIL-1971 Statement 
order of evaluation not per DFDL Spec

I'm going to suggest how we handle each of these for the 3.1.0 release.

I end up with 2 bugs to fix.

Keep in mind this is discussion fodder.

DAFFODIL-1422 disallow 
doctype decls in all XML & XSD that we read in

  *   release note it - try to fix for 3.2.0

DAFFODIL-2473 Need 
bit-wise AND, OR, NOT, and shift operations

  *   change to priority major - we need better documentation of the use case 
driving this.

DAFFODIL-2183 Unparse 
nilled complex element fails.

  *   Fix. This is actually a pretty bad bug, and some solutions to problems 
that have come up in real schemas need to use nilled complex elements in the 
solution.

DAFFODIL-1598 Unparser: 
For strings that truncate, the dfdl:valueLength function cannot suspend

  *Release Note, leave open. I question if this is critical. Change 
priority?

DAFFODIL-2399 Error 
diagnostics output even though there is an infoset

  *Fix. This is actually a pretty bad bug.

DAFFODIL-2400 New SAX API 
causes performance degredations.

  *   Release Note. Change priority to Major. I think this should not be 
critical as SAX is a relatively new feature, and while SAX is a 
performance-oriented feature, it's not like this is a regression on current 
functionality.

DAFFODIL-1971 Statement 
order of evaluation not per DFDL Spec

  *   Release Note.

Comments?


From: Interrante, John A (GE Research, US) 
Sent: Monday, April 12, 2021 2:52 PM
To: dev@daffodil.apache.org 
Subject: [Discuss] creating Release 3.1.0 and 96 JIRA tickets marked "Major" or 
higher

I found the list of open and in-progress issues sorted by priority 
(https://issues.apache.org/jira/issues/?jql=project%20%3D%20DAFFODIL%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC).
  I looked at all the critical issues (there are 8 in this list) and I don't 
think we have to hold up 3.1.0 for them although it would be nice if some got 
fixed first (e.g., DAFFODIL-1422 disallowing doctype decls for security and 
DAFFODIL-2400 fixing SAX API conformance and performance).

However, I would like to merge the new Runtime2 backend in time for the 3.1.0 
release of Daffodil in order to 1) accelerate development by avoiding the need 
to rebase the runtime2 branch on the master branch periodically, and 2) attract 
new developers to help me build out the runtime2 code further.  I need to 
finish a review of my outstanding pull request, merge it, rebase runtime2 on 
the master branch, submit a pull request to merge runtime2, and complete that 
PR’s review, so about 1-2 more weeks.

John

-Original Message-
From: Steve Lawrence 
Sent: Monday, April 12, 2021 1:33 PM
To: dev@daffodil.apache.org
Subject: EXT: Re: [Discuss] creating Release 3.1.0 and 96 JIRA tickets marked 
"Major" or higher

I would also like to see the updates to SAX merged--there's some important API 
conformance and performance fixes that are worth getting in. There's an open 
pull request (PR #478) that I think is close to being ready to be merged.

Of the open critical issues, I think DAFFODIL-2399 might be worth blocking if 
we were close to a fix. But I don't think anyone is actively working on it and 
I don't believe we ever tracked down what the issue was or what the right fix 
was. Hopefully that can be fixed by 3.2.0--I've seen that issue a few times and 
it's very confusing.

- Steve

On 4/12/21 1:01 PM, John Wass wrote:
>> I believe we need to do a release very soon regardless of these 96
>> issues
>
> I would like DAFFODIL- 2482 to get into it;
> https://github.com/apache/daffodil/pull/520
>
> Will increase priority on wrapping this

Re: The future of the daffodil DFDL schema debugger?

2021-04-20 Thread Beckerle, Mike 
Welcome Adam,

Here's the link to Adam's book, which looks very useful.

(Not shameless self promotion if someone else sends the link )

https://essentialeffects.dev/

-mikeb



From: Adam Rosien 
Sent: Monday, April 19, 2021 11:21 AM
To: dev@daffodil.apache.org 
Subject: Re: The future of the daffodil DFDL schema debugger?

Hi everybody, I've recently started working on Daffodil with some other
folks and will be helping where I can with the debugger.

I've been writing Scala since ~2011 and recently wrote a book about Cats
Effect, which has a similar scope to ZIO (effects, concurrency, etc.). If
anybody has any questions about the approach and techniques, I'm happy to
help.

.. Adam

Re: The future of the daffodil DFDL schema debugger?

2021-04-20 Thread John Wass 
> Next step is to refine these thoughts with a prototype.

Another next step is to collect feedback on this research and proposed
approach.  Any discussion is appreciated.



On Tue, Apr 20, 2021 at 10:00 AM John Wass  wrote:

> > Going to look deeper into how DAP might fit with Daffodil
>
> Have been looking over DAP and getting a good feeling about it. The
> specification [1] seems general enough that it could be applied to Daffodil
> and cover a swath of common operations (like start, stop, break, continue,
> code locations, variables, etc).
>
> There are many areas though that are unique to Daffodil that have no
> representation in the spec.  These things (like InputStream, Infoset, PoU,
> different variable types, backtracking, etc) will need an extension to
> DAP.  This really boils down to defining these things to fit under the DAP
> BaseProtocol and enabling handling of those objects on both the front and
> back ends.
>
> On the backend we need a Daffodil DAP protocol server.  Existing JVM
> implementations (like Java [2], Scala [3]) are tied closely to JDI and
> would bring a lot of extra baggage to work around that.  Developing a
> Daffodil specific implementation is no small task, but feasible.  There are
> a several existing implementations on the JVM that are close and can be
> looked at for reference.
>
> The backend implementation would look similar to what was described in an
> earlier post.  We could use ZIO/Akka/etc to implement the backend Protocol
> Server to enable the IO between the Daffodil process and the DAP clients.
> This implementation would now be guided by the DAP specification.
>
> With the protocol and backend extended to fit Daffodil that leaves the
> frontend.  In theory an existing IDE plugin should get pretty close to
> being able to perform the common debug operations mentioned above.  To
> support the Daffodil extensions there will need to be handling of the
> extended protocol into whatever views are desired/applicable.
>
> > Also looking into the Java Debug Interface (JDI) for comparison.
>
> JDI appears to be the wrong level of abstraction for what we are talking
> about in debugging Daffodil for schema development.  While DAP does do JVM
> debugging (through a JDI DAP impl) it also generalizes to many other
> debugging scenarios.  JDI on the other hand is very tied to the JVM.
>
> Extending the JDI appears to be more complex than dealing with DAP, and
> even though the JDI API is mostly defined with interfaces, there are choke
> points that limit to JVM concepts.  For example jdi.Value has a finite set
> of JVM types that it works with, its not clear where Daffodil types would
> plugin if even possible.
>
> The final note is that unique Daffodil features wouldn’t get to IDE
> support any faster JDI.  In some cases, like VS Code, you would still need
> an extended DAP to support these features.
>
> > and depending on how it shakes out will update the example to show
> integration
>
> It would appear wise to investigate DAP further.  Next step is to refine
> these thoughts with a prototype. I started an implementation in the example
> debugger project [4] to try to run the current example on a _minimal_ DAP
> implementation.
>
>
> [1] https://microsoft.github.io/debug-adapter-protocol/specification
> [2] https://github.com/Microsoft/java-debug
> [3] https://github.com/scalacenter/scala-debug-adapter
> [4] https://github.com/jw3/example-daffodil-debug
>
>
> On Mon, Apr 12, 2021 at 9:58 AM John Wass  wrote:
>
>> > the code is here https://github.com/jw3/example-daffodil-debug
>>
>> There is now a complete console based example for Zio that demonstrates
>> controlling the debug flow while distributing the current state to three
>> "displays".
>> 1. infoset at current step
>> 2. diff of infoset against previous step
>> 3. bit position and value of data.
>>
>> These displays are very rudimentary but demonstrate the ability to
>> asynchronously populate multiple views while synchronously controlling the
>> debug loop.
>>
>> > - The new protocol being informed by existing debugger and DAPis key
>>
>> Going to look deeper into how DAP might fit with Daffodil, and depending
>> on how it shakes out will update the example to show integration.
>>
>> Some interesting links to start with
>> - https://github.com/scalacenter/scala-debug-adapter
>> -
>> https://scalameta.org/metals/docs/integrations/debug-adapter-protocol.html
>> - https://github.com/microsoft/java-debug
>>
>> Also looking into the Java Debug Interface (JDI) for comparison.
>>
>>
>> On Thu, Apr 8, 2021 at 12:36 PM John Wass  wrote:
>>
>>> Revisiting this post after doing some debugger related work and thinking
>>> about debug protocol/adapters to connect external tooling to the debug
>>> process.
>>>
>>> This comment is good
>>>
>>> > This allo makes me wonder if an approach worth taking for the future
>>> of Daffodil schema debugging is developing a sort of "Daffodil Debug
>>> Protocol". I imagine it would be

Re: The future of the daffodil DFDL schema debugger?

2021-04-20 Thread John Wass 
> Going to look deeper into how DAP might fit with Daffodil

Have been looking over DAP and getting a good feeling about it. The
specification [1] seems general enough that it could be applied to Daffodil
and cover a swath of common operations (like start, stop, break, continue,
code locations, variables, etc).

There are many areas though that are unique to Daffodil that have no
representation in the spec.  These things (like InputStream, Infoset, PoU,
different variable types, backtracking, etc) will need an extension to
DAP.  This really boils down to defining these things to fit under the DAP
BaseProtocol and enabling handling of those objects on both the front and
back ends.

On the backend we need a Daffodil DAP protocol server.  Existing JVM
implementations (like Java [2], Scala [3]) are tied closely to JDI and
would bring a lot of extra baggage to work around that.  Developing a
Daffodil specific implementation is no small task, but feasible.  There are
a several existing implementations on the JVM that are close and can be
looked at for reference.

The backend implementation would look similar to what was described in an
earlier post.  We could use ZIO/Akka/etc to implement the backend Protocol
Server to enable the IO between the Daffodil process and the DAP clients.
This implementation would now be guided by the DAP specification.

With the protocol and backend extended to fit Daffodil that leaves the
frontend.  In theory an existing IDE plugin should get pretty close to
being able to perform the common debug operations mentioned above.  To
support the Daffodil extensions there will need to be handling of the
extended protocol into whatever views are desired/applicable.

> Also looking into the Java Debug Interface (JDI) for comparison.

JDI appears to be the wrong level of abstraction for what we are talking
about in debugging Daffodil for schema development.  While DAP does do JVM
debugging (through a JDI DAP impl) it also generalizes to many other
debugging scenarios.  JDI on the other hand is very tied to the JVM.

Extending the JDI appears to be more complex than dealing with DAP, and
even though the JDI API is mostly defined with interfaces, there are choke
points that limit to JVM concepts.  For example jdi.Value has a finite set
of JVM types that it works with, its not clear where Daffodil types would
plugin if even possible.

The final note is that unique Daffodil features wouldn’t get to IDE support
any faster JDI.  In some cases, like VS Code, you would still need an
extended DAP to support these features.

> and depending on how it shakes out will update the example to show
integration

It would appear wise to investigate DAP further.  Next step is to refine
these thoughts with a prototype. I started an implementation in the example
debugger project [4] to try to run the current example on a _minimal_ DAP
implementation.


[1] https://microsoft.github.io/debug-adapter-protocol/specification
[2] https://github.com/Microsoft/java-debug
[3] https://github.com/scalacenter/scala-debug-adapter
[4] https://github.com/jw3/example-daffodil-debug


On Mon, Apr 12, 2021 at 9:58 AM John Wass  wrote:

> > the code is here https://github.com/jw3/example-daffodil-debug
>
> There is now a complete console based example for Zio that demonstrates
> controlling the debug flow while distributing the current state to three
> "displays".
> 1. infoset at current step
> 2. diff of infoset against previous step
> 3. bit position and value of data.
>
> These displays are very rudimentary but demonstrate the ability to
> asynchronously populate multiple views while synchronously controlling the
> debug loop.
>
> > - The new protocol being informed by existing debugger and DAPis key
>
> Going to look deeper into how DAP might fit with Daffodil, and depending
> on how it shakes out will update the example to show integration.
>
> Some interesting links to start with
> - https://github.com/scalacenter/scala-debug-adapter
> -
> https://scalameta.org/metals/docs/integrations/debug-adapter-protocol.html
> - https://github.com/microsoft/java-debug
>
> Also looking into the Java Debug Interface (JDI) for comparison.
>
>
> On Thu, Apr 8, 2021 at 12:36 PM John Wass  wrote:
>
>> Revisiting this post after doing some debugger related work and thinking
>> about debug protocol/adapters to connect external tooling to the debug
>> process.
>>
>> This comment is good
>>
>> > This allo makes me wonder if an approach worth taking for the future of
>> Daffodil schema debugging is developing a sort of "Daffodil Debug
>> Protocol". I imagine it would be loosely based on DAP (which is
>> essentially JSON message based) but could be targeted to the things that a
>> DFDL schema debugger would really need. An added benefit with some  sort of
>> protocol is the debugger interface can be uncoupled from Daffodil
>> itself, so we could implement a TUI/GUI/whatever in any  language/GUI
>> framework and just have it communicate the protocol