Re: The future of the daffodil DFDL schema debugger?
This is actually very cool using ZIO for this. I have to learn more about ZIO. From: John Wass Sent: Monday, April 12, 2021 9:58 AM To: dev@daffodil.apache.org Subject: Re: The future of the daffodil DFDL schema debugger? > the code is here https://github.com/jw3/example-daffodil-debug There is now a complete console based example for Zio that demonstrates controlling the debug flow while distributing the current state to three "displays". 1. infoset at current step 2. diff of infoset against previous step 3. bit position and value of data. These displays are very rudimentary but demonstrate the ability to asynchronously populate multiple views while synchronously controlling the debug loop. > - The new protocol being informed by existing debugger and DAPis key Going to look deeper into how DAP might fit with Daffodil, and depending on how it shakes out will update the example to show integration. Some interesting links to start with - https://github.com/scalacenter/scala-debug-adapter - https://scalameta.org/metals/docs/integrations/debug-adapter-protocol.html - https://github.com/microsoft/java-debug Also looking into the Java Debug Interface (JDI) for comparison. On Thu, Apr 8, 2021 at 12:36 PM John Wass wrote: > Revisiting this post after doing some debugger related work and thinking > about debug protocol/adapters to connect external tooling to the debug > process. > > This comment is good > > > This allo makes me wonder if an approach worth taking for the future of > Daffodil schema debugging is developing a sort of "Daffodil Debug Protocol". > I imagine it would be loosely based on DAP (which is essentially JSON > message based) but could be targeted to the things that a DFDL schema > debugger would really need. An added benefit with some sort of protocol > is the debugger interface can be uncoupled from Daffodil itself, so we > could implement a TUI/GUI/whatever in any language/GUI framework and just > have it communicate the protocol over some form of IPC. Another benefit > is that any future backends could implement this protocol and so a single > debugger could hook into different backends without much issue. > Unfortunately, defining such a protocol might be a large task, but we do > have our existing debug infrastructure and things like DAP to guide its > development/design. > > Some thoughts on this > - Defining the protocol will be a large task, but a minimal version should > get up and round tripping quickly with a minimal subset of the protocol. > - The new protocol being informed by existing debugger and DAPis key > - Uncoupling from Daffodil is key > - Adapt the Daffodil protocol to produce DAP after the fact so as not to > constrain Daffodil debugging capability > - We dont need to tie the protocol or adapters to a single framework, > implementations of the IO layer should be simple enough to support multiple > things (eg Akka, Zio, "basic" ...) > - The current debugger lives in runtime1, but can we make an abstract API > that any runtime would implement? > > Maybe a solution is structured like this > - daffodil-debug-api: > - protocol model > - interfaces: debugger / IO adapter / etc > - lives in daffodil repo (new subproject?) > - daffodil-debug-io-NAME > - provides implementation of a specific IO adapter > - multiple projects possible (daffodil-debugger-akka, > daffodil-debugger-zio, etc) > - supported ones live in their own subprojects, but other can be plugged > in from external sources > - ability to support multiple implementations reduces risk of lock-in > - debugger applications > - maintained in external repositories > - depending on the IO implementation these could execute be in separate > process or on separate machine > - like Steve said, could be any language / framework > > Three types of reference implementations / sample applications could also > guide the development of the API > 1. a replacement for the existing TUI debugger, expected to end up with > at minimum the same functionality as the current one. > 2. a standalone GUI (JavaFX, Scala.js, ..) debugger > 3. an IDE integration > > Thoughts? > > Also I'm working on some reference implementations of these concepts using > Akka and Zio. Not quite ready to talk through it yet, but the code is here > https://github.com/jw3/example-daffodil-debug > > > > On Wed, Jan 6, 2021 at 1:42 PM Steve Lawrence > wrote: > >> Yep, something like that seems very reasonable for dealing with large >> infosets. But it still feels like we still run into usability issues. >> For example, what if a user wants to see more? We need some >> configuration options to increase what we've ellided. It's not big, but >> every new thing that needs configuration adds complexity and decreases >> usability. >> >> And I think the only reason we are trying to spend effort elliding >> things is because we're limited to this gdb-like interface where you can >> only print out a
RE: [daffodil-site] branch asf-site updated: Publishing from 64abbf0df41a006a7a19d7daad29749b0a7dd6b5
@@ -155,8 +193,7 @@ and structuring our comments that way.
DSOM "fixed" getter
Note: If we change runtime1 to validate "fixed" values
-like runtime2 does, then
-we can close https://issues.apache.org/jira/browse/DAFFODIL-117;>DAFFODIL-117.
+like runtime2 does, then we can resolve {% jira 117 %}.
So much for hoping that the site generator supports the markdown jira macro :).
https://daffodil.apache.org/dev/design-notes/runtime2-todos/
-Original Message-
From: github-...@apache.org
Sent: Friday, April 16, 2021 10:57 AM
To: comm...@daffodil.apache.org
Subject: EXT: [daffodil-site] branch asf-site updated: Publishing from
64abbf0df41a006a7a19d7daad29749b0a7dd6b5
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site in repository
https://gitbox.apache.org/repos/asf/daffodil-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new fc38a6c Publishing from 64abbf0df41a006a7a19d7daad29749b0a7dd6b5
fc38a6c is described below
commit fc38a6c0ac407b77f77b222f9270c53509055f5b
Author: John Interrante
AuthorDate: Fri Apr 16 14:57:03 2021 +
Publishing from 64abbf0df41a006a7a19d7daad29749b0a7dd6b5
---
content/dev/design-notes/runtime2-todos/index.html | 81 --
1 file changed, 59 insertions(+), 22 deletions(-)
diff --git a/content/dev/design-notes/runtime2-todos/index.html
b/content/dev/design-notes/runtime2-todos/index.html
index 184f6be..362c979 100644
--- a/content/dev/design-notes/runtime2-todos/index.html
+++ b/content/dev/design-notes/runtime2-todos/index.html
@@ -113,34 +113,72 @@ in order to avoid duplication.
-Error struct instead of error
message
+Reporting errors
+using structs, not strings
-This change is almost complete.
-We have replaced error message strings
+We have replaced error message strings
with error structs everywhere now.
-However, we should support passing
-a schema location and/or data position
-through the error struct as well as
-an error code, an integer, and a string.
-We may need to store schema locations
-(filenames and line numbers) in ERD objects -and pass an ERD in the error
struct.
+However, we may need to expand the error struct to include a pointer
+(pstate/ustate for data position) and another pointer (ERD or static
+context object for schema filename/line number). We also may want to implement error logging
+variants that both do and dont humanize the errors, e.g., a
+hardware/FPGA-type implementation might just output numbers and an
+external tool might have to "humanize" these numbers using knowledge of
+the schema and runtime data objects, like an offline log processor
+does.
-Different types of errors
+Recovering after errors
-When runtime2 grows larger,
-we may need to distinguish handle -more types of errors.
-Right now we handle only
-fatal errors and validation errors.
-Fatal errors stop the program immediately.
+As we continue to build out runtime2, we may need to distinguish
+more types of errors and allow backtracking and retrying.
+Right now we handle only parse/unparse and validation errors in limited
+ways.
+Parse/unparse errors abort the parsing/unparsing and return to the
+caller immediately without resetting the streams position.
Validation errors are collected in an array and printed after parsing or
unparsing.
-Later we may need to handle
-parser/unparser warnings and errors
-and allow backtracking after these errors.
+The only places where there are calls to stop the program are in
+daffodil_main.c (top-level error handling) and stack.c (empty,
+overflow, underflow errors which should never happen). Most of the parse functions set
+pstateerror only if they couldnt read data into their
+buffer due to an I/O error or EOF, which doesnt seem recoverable
+to me.
+Likewise, the unparse functions set ustateerror only if they
+couldnt write data from their buffer due to an I/O error, which
+doesnt seem recoverable to me. Only the parse_endian_bool functions set
+pstateerror if they read an integer which doesnt match
+either true_rep or false_rep when an exact match to either is required.
+If we decide to implement backtracking and retrying, they should call
+fseek to reset the streams position back to where they started
+reading the integer before they return to their callers.
+Right now all parse calls are followed by if statements to check for
+error and return immediately.
+The code generator would have to generate code which can advance the
+streams position by some byte(s) and try the parse call again as
+an attempt to resynchronize with a correct data stream after a bunch of
+failures. Note that we actually
+run the generated code in an embedded processor and call our own
+fread/frwrite functions which replace the stdio fread/fwrite functions
+since the C code runs bare metal without OS functions.
+We can implement fseek but we should have a good use
