[Discuss] creating Release 3.1.0 and 96 JIRA tickets marked "Major" or higher
I found the list of open and in-progress issues sorted by priority (https://issues.apache.org/jira/issues/?jql=project%20%3D%20DAFFODIL%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC). I looked at all the critical issues (there are 8 in this list) and I don't think we have to hold up 3.1.0 for them although it would be nice if some got fixed first (e.g., DAFFODIL-1422 disallowing doctype decls for security and DAFFODIL-2400 fixing SAX API conformance and performance). However, I would like to merge the new Runtime2 backend in time for the 3.1.0 release of Daffodil in order to 1) accelerate development by avoiding the need to rebase the runtime2 branch on the master branch periodically, and 2) attract new developers to help me build out the runtime2 code further. I need to finish a review of my outstanding pull request, merge it, rebase runtime2 on the master branch, submit a pull request to merge runtime2, and complete that PR’s review, so about 1-2 more weeks. John -Original Message- From: Steve Lawrence Sent: Monday, April 12, 2021 1:33 PM To: dev@daffodil.apache.org Subject: EXT: Re: [Discuss] creating Release 3.1.0 and 96 JIRA tickets marked "Major" or higher I would also like to see the updates to SAX merged--there's some important API conformance and performance fixes that are worth getting in. There's an open pull request (PR #478) that I think is close to being ready to be merged. Of the open critical issues, I think DAFFODIL-2399 might be worth blocking if we were close to a fix. But I don't think anyone is actively working on it and I don't believe we ever tracked down what the issue was or what the right fix was. Hopefully that can be fixed by 3.2.0--I've seen that issue a few times and it's very confusing. - Steve On 4/12/21 1:01 PM, John Wass wrote: >> I believe we need to do a release very soon regardless of these 96 >> issues > > I would like DAFFODIL- 2482 to get into it; > https://github.com/apache/daffodil/pull/520 > > Will increase priority on wrapping this up. > > > > On Mon, Apr 12, 2021 at 12:43 PM Beckerle, Mike < > mbecke...@owlcyberdefense.com> wrote: > >> I'd like to discuss our need to create a new release of Daffodil, >> which would be 3.1.0. >> >> We have added enough new functionality certainly to justify a release. >> There are important features already complete, there is the new >> Runtime2 backend, etc. >> >> The challenge is that we have 96 JIRA tickets specifically for bugs >> that are marked "major" or above in priority. 6 are marked critical, >> so 90 are "major". (I am excluding all "improvement" and >> "new-feature" tickets in this count. Just bugs.) Obviously we're not >> going to fix 96 issues super quickly. >> >> Some people advocate a set of criteria for releases which stipulate >> there can be no critical/blocker issues, and no major issues, only minor >> issues. >> However, the status of critical/major/minor on our JIRA tickets is >> subjective, most bugs are found and reported by us. >> >> Exactly two bugs have "votes" more than zero, which reflects that >> we've not been using the votes field to prioritize anything, but >> perhaps we should use votes moving forward, rather than bumping >> priorities up and down based on our subjective assessment of importance. >> >> I believe we need to do a release very soon regardless of these 96 issues. >> In scrolling through them, evaluating them as "are they more >> important than doing our first TLP release", none of them rise to >> that level of importance to me. >> >> Most of these issues are part of release 3.0.0 and before that as >> well, so >> 3.1.0 would still be an improvement. >> >> One way to deal with the critical issues is to specifically discuss >> them in a release note. >> >> Please let's discuss openly. What do you believe must be in 3.1.0, >> that we would hold up a release over? >> >> -mike beckerle >> >> >> >
Re: [Discuss] creating Release 3.1.0 and 96 JIRA tickets marked "Major" or higher
I would also like to see the updates to SAX merged--there's some important API conformance and performance fixes that are worth getting in. There's an open pull request (PR #478) that I think is close to being ready to be merged. Of the open critical issues, I think DAFFODIL-2399 might be worth blocking if we were close to a fix. But I don't think anyone is actively working on it and I don't believe we ever tracked down what the issue was or what the right fix was. Hopefully that can be fixed by 3.2.0--I've seen that issue a few times and it's very confusing. - Steve On 4/12/21 1:01 PM, John Wass wrote: >> I believe we need to do a release very soon regardless of these 96 issues > > I would like DAFFODIL- 2482 to get into it; > https://github.com/apache/daffodil/pull/520 > > Will increase priority on wrapping this up. > > > > On Mon, Apr 12, 2021 at 12:43 PM Beckerle, Mike < > mbecke...@owlcyberdefense.com> wrote: > >> I'd like to discuss our need to create a new release of Daffodil, which >> would be 3.1.0. >> >> We have added enough new functionality certainly to justify a release. >> There are important features already complete, there is the new Runtime2 >> backend, etc. >> >> The challenge is that we have 96 JIRA tickets specifically for bugs that >> are marked "major" or above in priority. 6 are marked critical, so 90 are >> "major". (I am excluding all "improvement" and "new-feature" tickets in >> this count. Just bugs.) Obviously we're not going to fix 96 issues super >> quickly. >> >> Some people advocate a set of criteria for releases which stipulate there >> can be no critical/blocker issues, and no major issues, only minor issues. >> However, the status of critical/major/minor on our JIRA tickets is >> subjective, most bugs are found and reported by us. >> >> Exactly two bugs have "votes" more than zero, which reflects that we've >> not been using the votes field to prioritize anything, but perhaps we >> should use votes moving forward, rather than bumping priorities up and down >> based on our subjective assessment of importance. >> >> I believe we need to do a release very soon regardless of these 96 issues. >> In scrolling through them, evaluating them as "are they more important than >> doing our first TLP release", none of them rise to that level of importance >> to me. >> >> Most of these issues are part of release 3.0.0 and before that as well, so >> 3.1.0 would still be an improvement. >> >> One way to deal with the critical issues is to specifically discuss them >> in a release note. >> >> Please let's discuss openly. What do you believe must be in 3.1.0, that >> we would hold up a release over? >> >> -mike beckerle >> >> >> >
Re: [Discuss] creating Release 3.1.0 and 96 JIRA tickets marked "Major" or higher
> I believe we need to do a release very soon regardless of these 96 issues I would like DAFFODIL- 2482 to get into it; https://github.com/apache/daffodil/pull/520 Will increase priority on wrapping this up. On Mon, Apr 12, 2021 at 12:43 PM Beckerle, Mike < mbecke...@owlcyberdefense.com> wrote: > I'd like to discuss our need to create a new release of Daffodil, which > would be 3.1.0. > > We have added enough new functionality certainly to justify a release. > There are important features already complete, there is the new Runtime2 > backend, etc. > > The challenge is that we have 96 JIRA tickets specifically for bugs that > are marked "major" or above in priority. 6 are marked critical, so 90 are > "major". (I am excluding all "improvement" and "new-feature" tickets in > this count. Just bugs.) Obviously we're not going to fix 96 issues super > quickly. > > Some people advocate a set of criteria for releases which stipulate there > can be no critical/blocker issues, and no major issues, only minor issues. > However, the status of critical/major/minor on our JIRA tickets is > subjective, most bugs are found and reported by us. > > Exactly two bugs have "votes" more than zero, which reflects that we've > not been using the votes field to prioritize anything, but perhaps we > should use votes moving forward, rather than bumping priorities up and down > based on our subjective assessment of importance. > > I believe we need to do a release very soon regardless of these 96 issues. > In scrolling through them, evaluating them as "are they more important than > doing our first TLP release", none of them rise to that level of importance > to me. > > Most of these issues are part of release 3.0.0 and before that as well, so > 3.1.0 would still be an improvement. > > One way to deal with the critical issues is to specifically discuss them > in a release note. > > Please let's discuss openly. What do you believe must be in 3.1.0, that > we would hold up a release over? > > -mike beckerle > > >
[Discuss] creating Release 3.1.0 and 96 JIRA tickets marked "Major" or higher
I'd like to discuss our need to create a new release of Daffodil, which would be 3.1.0. We have added enough new functionality certainly to justify a release. There are important features already complete, there is the new Runtime2 backend, etc. The challenge is that we have 96 JIRA tickets specifically for bugs that are marked "major" or above in priority. 6 are marked critical, so 90 are "major". (I am excluding all "improvement" and "new-feature" tickets in this count. Just bugs.) Obviously we're not going to fix 96 issues super quickly. Some people advocate a set of criteria for releases which stipulate there can be no critical/blocker issues, and no major issues, only minor issues. However, the status of critical/major/minor on our JIRA tickets is subjective, most bugs are found and reported by us. Exactly two bugs have "votes" more than zero, which reflects that we've not been using the votes field to prioritize anything, but perhaps we should use votes moving forward, rather than bumping priorities up and down based on our subjective assessment of importance. I believe we need to do a release very soon regardless of these 96 issues. In scrolling through them, evaluating them as "are they more important than doing our first TLP release", none of them rise to that level of importance to me. Most of these issues are part of release 3.0.0 and before that as well, so 3.1.0 would still be an improvement. One way to deal with the critical issues is to specifically discuss them in a release note. Please let's discuss openly. What do you believe must be in 3.1.0, that we would hold up a release over? -mike beckerle
Re: The future of the daffodil DFDL schema debugger?
> the code is here https://github.com/jw3/example-daffodil-debug There is now a complete console based example for Zio that demonstrates controlling the debug flow while distributing the current state to three "displays". 1. infoset at current step 2. diff of infoset against previous step 3. bit position and value of data. These displays are very rudimentary but demonstrate the ability to asynchronously populate multiple views while synchronously controlling the debug loop. > - The new protocol being informed by existing debugger and DAPis key Going to look deeper into how DAP might fit with Daffodil, and depending on how it shakes out will update the example to show integration. Some interesting links to start with - https://github.com/scalacenter/scala-debug-adapter - https://scalameta.org/metals/docs/integrations/debug-adapter-protocol.html - https://github.com/microsoft/java-debug Also looking into the Java Debug Interface (JDI) for comparison. On Thu, Apr 8, 2021 at 12:36 PM John Wass wrote: > Revisiting this post after doing some debugger related work and thinking > about debug protocol/adapters to connect external tooling to the debug > process. > > This comment is good > > > This allo makes me wonder if an approach worth taking for the future of > Daffodil schema debugging is developing a sort of "Daffodil Debug Protocol". > I imagine it would be loosely based on DAP (which is essentially JSON > message based) but could be targeted to the things that a DFDL schema > debugger would really need. An added benefit with some sort of protocol > is the debugger interface can be uncoupled from Daffodil itself, so we > could implement a TUI/GUI/whatever in any language/GUI framework and just > have it communicate the protocol over some form of IPC. Another benefit > is that any future backends could implement this protocol and so a single > debugger could hook into different backends without much issue. > Unfortunately, defining such a protocol might be a large task, but we do > have our existing debug infrastructure and things like DAP to guide its > development/design. > > Some thoughts on this > - Defining the protocol will be a large task, but a minimal version should > get up and round tripping quickly with a minimal subset of the protocol. > - The new protocol being informed by existing debugger and DAPis key > - Uncoupling from Daffodil is key > - Adapt the Daffodil protocol to produce DAP after the fact so as not to > constrain Daffodil debugging capability > - We dont need to tie the protocol or adapters to a single framework, > implementations of the IO layer should be simple enough to support multiple > things (eg Akka, Zio, "basic" ...) > - The current debugger lives in runtime1, but can we make an abstract API > that any runtime would implement? > > Maybe a solution is structured like this > - daffodil-debug-api: > - protocol model > - interfaces: debugger / IO adapter / etc > - lives in daffodil repo (new subproject?) > - daffodil-debug-io-NAME > - provides implementation of a specific IO adapter > - multiple projects possible (daffodil-debugger-akka, > daffodil-debugger-zio, etc) > - supported ones live in their own subprojects, but other can be plugged > in from external sources > - ability to support multiple implementations reduces risk of lock-in > - debugger applications > - maintained in external repositories > - depending on the IO implementation these could execute be in separate > process or on separate machine > - like Steve said, could be any language / framework > > Three types of reference implementations / sample applications could also > guide the development of the API > 1. a replacement for the existing TUI debugger, expected to end up with > at minimum the same functionality as the current one. > 2. a standalone GUI (JavaFX, Scala.js, ..) debugger > 3. an IDE integration > > Thoughts? > > Also I'm working on some reference implementations of these concepts using > Akka and Zio. Not quite ready to talk through it yet, but the code is here > https://github.com/jw3/example-daffodil-debug > > > > On Wed, Jan 6, 2021 at 1:42 PM Steve Lawrence > wrote: > >> Yep, something like that seems very reasonable for dealing with large >> infosets. But it still feels like we still run into usability issues. >> For example, what if a user wants to see more? We need some >> configuration options to increase what we've ellided. It's not big, but >> every new thing that needs configuration adds complexity and decreases >> usability. >> >> And I think the only reason we are trying to spend effort elliding >> things is because we're limited to this gdb-like interface where you can >> only print out a little information at a time. >> >> I think what would really is to dump this gdb interface and instead use >> multiple windows/views. As a really close example to what I imagine, I >> recently came across this hex editor: >> >> https://www.synalysis.net/ >>
