Re: Daffodil DAP debugger modules and repos
Thanks! On Tue, May 4, 2021 at 3:07 PM Beckerle, Mike wrote: > Grrr. I can't write to it however. > > INFRA says it may just be an hour before the permissions propagate to it. > > I hope to update this Wednesday, push a single file over there so people > can fork it and get started. > > I confirmed vscode is MIT license, which is Category A, as in "allowed". > > > > > > From: Beckerle, Mike > Sent: Tuesday, May 4, 2021 5:46 PM > To: dev@daffodil.apache.org > Subject: Re: Daffodil DAP debugger modules and repos > > Well I'd like to see this be in an Apache Daffodil repo. > > In fact, I just created one. You can find it at > > https://github.com/apache/daffodil-vscode > > The DFDLSchemas is not directly analogous, as there are other DFDL > implementations and numerous schemas there are created by others for use > with those implementations. E.g., EDIFACT, iso8586, etc. It also all > significantly pre-dates Apache Daffodil. > > > > > > From: Adam Rosien > Sent: Tuesday, May 4, 2021 5:36 PM > To: dev@daffodil.apache.org > Subject: Daffodil DAP debugger modules and repos > > I've been extending John's debugger prototype to support DAP, the debug > protocol supported by VS Code and other IDEs. There's an animated gif of > what the VS Code interaction looks like, where only the current schema > element and data position are relayed, at [1]. > > Now that we've made these first steps, we wanted the community's advice and > opinion about where the related code should live. Here is our initial > proposal: > > * The VS Code extension would live in a separate repository, > `daffodil-vscode`. This is a common pattern with other extensions, and > would allow the extension to be released independently of daffodil itself. > However, I'm not sure what "organization" this would live under; this > situation is similar to auxiliary Daffodil repos like > https://github.com/DFDLSchemas. > > * The Daffodil `Debugger` to DAP code *could* exist as a sub-module of the > main Daffodil project, say, `daffodil-dap`. We expect a lot of churn for > this code as we translate more and more of the Daffodil parsing state into > the DAP domain. There are a few new dependencies, like the java-debug > project that handles the DAP protocol, and helper code like cats and fs2 > (for streaming). > > That's the basics. We'd love to know if this fits or if you have some > better ideas. > > .. Adam > > [1] https://github.com/jw3/example-daffodil-debug/discussions/10 >
Re: Daffodil DAP debugger modules and repos
Grrr. I can't write to it however. INFRA says it may just be an hour before the permissions propagate to it. I hope to update this Wednesday, push a single file over there so people can fork it and get started. I confirmed vscode is MIT license, which is Category A, as in "allowed". From: Beckerle, Mike Sent: Tuesday, May 4, 2021 5:46 PM To: dev@daffodil.apache.org Subject: Re: Daffodil DAP debugger modules and repos Well I'd like to see this be in an Apache Daffodil repo. In fact, I just created one. You can find it at https://github.com/apache/daffodil-vscode The DFDLSchemas is not directly analogous, as there are other DFDL implementations and numerous schemas there are created by others for use with those implementations. E.g., EDIFACT, iso8586, etc. It also all significantly pre-dates Apache Daffodil. From: Adam Rosien Sent: Tuesday, May 4, 2021 5:36 PM To: dev@daffodil.apache.org Subject: Daffodil DAP debugger modules and repos I've been extending John's debugger prototype to support DAP, the debug protocol supported by VS Code and other IDEs. There's an animated gif of what the VS Code interaction looks like, where only the current schema element and data position are relayed, at [1]. Now that we've made these first steps, we wanted the community's advice and opinion about where the related code should live. Here is our initial proposal: * The VS Code extension would live in a separate repository, `daffodil-vscode`. This is a common pattern with other extensions, and would allow the extension to be released independently of daffodil itself. However, I'm not sure what "organization" this would live under; this situation is similar to auxiliary Daffodil repos like https://github.com/DFDLSchemas. * The Daffodil `Debugger` to DAP code *could* exist as a sub-module of the main Daffodil project, say, `daffodil-dap`. We expect a lot of churn for this code as we translate more and more of the Daffodil parsing state into the DAP domain. There are a few new dependencies, like the java-debug project that handles the DAP protocol, and helper code like cats and fs2 (for streaming). That's the basics. We'd love to know if this fits or if you have some better ideas. .. Adam [1] https://github.com/jw3/example-daffodil-debug/discussions/10
Re: Daffodil DAP debugger modules and repos
Well I'd like to see this be in an Apache Daffodil repo. In fact, I just created one. You can find it at https://github.com/apache/daffodil-vscode The DFDLSchemas is not directly analogous, as there are other DFDL implementations and numerous schemas there are created by others for use with those implementations. E.g., EDIFACT, iso8586, etc. It also all significantly pre-dates Apache Daffodil. From: Adam Rosien Sent: Tuesday, May 4, 2021 5:36 PM To: dev@daffodil.apache.org Subject: Daffodil DAP debugger modules and repos I've been extending John's debugger prototype to support DAP, the debug protocol supported by VS Code and other IDEs. There's an animated gif of what the VS Code interaction looks like, where only the current schema element and data position are relayed, at [1]. Now that we've made these first steps, we wanted the community's advice and opinion about where the related code should live. Here is our initial proposal: * The VS Code extension would live in a separate repository, `daffodil-vscode`. This is a common pattern with other extensions, and would allow the extension to be released independently of daffodil itself. However, I'm not sure what "organization" this would live under; this situation is similar to auxiliary Daffodil repos like https://github.com/DFDLSchemas. * The Daffodil `Debugger` to DAP code *could* exist as a sub-module of the main Daffodil project, say, `daffodil-dap`. We expect a lot of churn for this code as we translate more and more of the Daffodil parsing state into the DAP domain. There are a few new dependencies, like the java-debug project that handles the DAP protocol, and helper code like cats and fs2 (for streaming). That's the basics. We'd love to know if this fits or if you have some better ideas. .. Adam [1] https://github.com/jw3/example-daffodil-debug/discussions/10
Daffodil DAP debugger modules and repos
I've been extending John's debugger prototype to support DAP, the debug protocol supported by VS Code and other IDEs. There's an animated gif of what the VS Code interaction looks like, where only the current schema element and data position are relayed, at [1]. Now that we've made these first steps, we wanted the community's advice and opinion about where the related code should live. Here is our initial proposal: * The VS Code extension would live in a separate repository, `daffodil-vscode`. This is a common pattern with other extensions, and would allow the extension to be released independently of daffodil itself. However, I'm not sure what "organization" this would live under; this situation is similar to auxiliary Daffodil repos like https://github.com/DFDLSchemas. * The Daffodil `Debugger` to DAP code *could* exist as a sub-module of the main Daffodil project, say, `daffodil-dap`. We expect a lot of churn for this code as we translate more and more of the Daffodil parsing state into the DAP domain. There are a few new dependencies, like the java-debug project that handles the DAP protocol, and helper code like cats and fs2 (for streaming). That's the basics. We'd love to know if this fits or if you have some better ideas. .. Adam [1] https://github.com/jw3/example-daffodil-debug/discussions/10
Re: The future of the daffodil DFDL schema debugger?
> It looks like some of the code from java-debug can be reused without involving JDI. The java-debug project could be viewed as an implementation of the DAP communication protocol, coupled with JDI to provide request/response values to DAP. For example, the `ProtocolServer` [3] hard-codes the JDI, but there's an `AbstractProtocolServer` which only handles the DAP communication (as a rough guess). Update: I've successfully used the java-debug project from Microsoft to speak DAP without requiring any JDI dependencies, for use in integrating with the Daffodil Debugger interface. On Mon, Apr 26, 2021 at 10:23 AM Adam Rosien wrote: > I'm currently seeing what it takes to get a minimal VS Code extension > talking DAP over stdin/stdout to an external Scala process. > > On Fri, Apr 23, 2021 at 11:01 AM Adam Rosien wrote: > >> I've looked at scala-debug-adapter a bit now, and it doesn't do very >> much: there's some socket stuff and state management, but otherwise it >> delegates to the underflying java-debug library which manages the DAP >> protocol [1]. *That* library does assume use of JDI and supplies JVM-level >> stuff to DAP (threads, etc.). >> >> So I think we don't want to rely on the code directly, but could extract >> the outer "skeleton" of `DebugServer` [2] to use with Daffodil. >> >> It looks like some of the code from java-debug can be reused without >> involving JDI. The java-debug project could be viewed as an implementation >> of the DAP communication protocol, coupled with JDI to provide >> request/response values to DAP. For example, the `ProtocolServer` [3] >> hard-codes the JDI, but there's an `AbstractProtocolServer` which only >> handles the DAP communication (as a rough guess). >> >> I think the next step is to play with the library in the prototype repo >> to see what is really needed. >> >> .. Adam >> >> [1] >> https://github.com/scalacenter/scala-debug-adapter/blob/main/core/src/main/scala/ch/epfl/scala/debugadapter/internal/DebugSession.scala#L35 >> extends java-debug `ProtocolServer`. >> [2] >> https://github.com/scalacenter/scala-debug-adapter/blob/main/core/src/main/scala/ch/epfl/scala/debugadapter/DebugServer.scala >> [3] >> https://github.com/microsoft/java-debug/blob/master/com.microsoft.java.debug.core/src/main/java/com/microsoft/java/debug/core/adapter/ProtocolServer.java#L52 >> >> On Thu, Apr 22, 2021 at 1:31 PM John Wass wrote: >> >>> > dig a bit to see if the DAP-only hooks can be reused without JDI coming >>> along for the ride >>> >>> Cool, that would be good to dig at. Big win if we can reuse it. >>> >>
Re: Escape character parsing bug?
I'll begin making the change to add an SDE for these then. It seems that most of the escape scheme tests that weren't round tripping were cases like this. Josh On May 4, 2021 12:15 PM, "Beckerle, Mike" wrote: I asked Steve Hanson of IBM - other co-chair on DFDL workgroup, and one of the primaries on one of IBM's DFDL implementations, said that when he tries this situation with the escape character "/" matching the start of the separator, he gets an SDE. It appears not to be part of the DFDL spec to call this out as an SDE, so that omission will likely become the first erratum to the DFDL v1.0 official final spec. From: Adams, Joshua Sent: Monday, May 3, 2021 3:35 PM To: dev@daffodil.apache.org Subject: Re: Escape character parsing bug? Thanks for running this up the chain so to speak. I agree that an SDE would probably be best for situations like this as I wouldn't think any sort of sane data format would use a combination of separators/escape characters like this. Josh From: Beckerle, Mike Sent: Monday, May 3, 2021 3:32 PM To: dev@daffodil.apache.org Subject: Re: Escape character parsing bug? So you have a separator the first char of which is the escape character. Yikes. I think the DFDL spec should, ideally, make this an SDE. Feels entirely ambiguous to me. The part of the spec you quote is quite problematic, but was updated by one word in the final DFDL Spec version. Occurrences of the dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are removed from the data, unless the dfdl:escapeCharacter is preceded by the dfdl:escapeEscapeCharacter, or the dfdl:escapeEscapeCharacter does not precede the dfdl:escapeCharacter, respectively. So breaking that into two independent statements: 1. An escapeCharacter is removed unless it is preceded by the escape-escape. 2. An escape-escape is removed unless it does not precede the escape character. So (1) means an escape char that is floating around not in front of any delimiter is removed. (2) means an escape-escape floating around not in front of any escape char, is preserved. That still doesn't help with your specific issue. If a delimiter begins with the escapeCharacter, will that delimiter appearing in the data be interpreted as an escape character followed by the 2nd and subsequent characters of the delimiter? Or will the delimiter be recognized? Consider dfdl:separator="/ // ///" with escapeCharacter="/" and escapeEscapeCharacter="/" What takes priority, interpretation of escapeCharacters and escapeEscapeCharacters or recognizing delimiters? I have posed this issue for consideration of the other DFDL workgroup members and I'll report back. From: Adams, Joshua Sent: Monday, May 3, 2021 2:38 PM To: dev@daffodil.apache.org Subject: Escape character parsing bug? Consider the following schema: We then have the following test case: foo$$/;bar foo$/;bar Shouldn't this parse as: foo$$ bar The spec says the following: On parsing any in-scope terminating delimiter encountered in the data is not interpreted as such when it is immediately preceded by the dfdl:escapeCharacter (when not itself preceded by the dfdl:escapeEscapeCharacter). Occurrences of the dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are removed from the data, unless the dfdl:escapeCharacter is preceded by the dfdl:escapeEscapeCharacter, or the dfdl:escapeEscapeCharacter does not precede the dfdl:escapeCharacter. It seems to me that the '/;' terminator shouldn't be getting escaped in this case, but want to double check. Josh
Re: Escape character parsing bug?
I asked Steve Hanson of IBM - other co-chair on DFDL workgroup, and one of the primaries on one of IBM's DFDL implementations, said that when he tries this situation with the escape character "/" matching the start of the separator, he gets an SDE. It appears not to be part of the DFDL spec to call this out as an SDE, so that omission will likely become the first erratum to the DFDL v1.0 official final spec. From: Adams, Joshua Sent: Monday, May 3, 2021 3:35 PM To: dev@daffodil.apache.org Subject: Re: Escape character parsing bug? Thanks for running this up the chain so to speak. I agree that an SDE would probably be best for situations like this as I wouldn't think any sort of sane data format would use a combination of separators/escape characters like this. Josh From: Beckerle, Mike Sent: Monday, May 3, 2021 3:32 PM To: dev@daffodil.apache.org Subject: Re: Escape character parsing bug? So you have a separator the first char of which is the escape character. Yikes. I think the DFDL spec should, ideally, make this an SDE. Feels entirely ambiguous to me. The part of the spec you quote is quite problematic, but was updated by one word in the final DFDL Spec version. Occurrences of the dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are removed from the data, unless the dfdl:escapeCharacter is preceded by the dfdl:escapeEscapeCharacter, or the dfdl:escapeEscapeCharacter does not precede the dfdl:escapeCharacter, respectively. So breaking that into two independent statements: 1. An escapeCharacter is removed unless it is preceded by the escape-escape. 2. An escape-escape is removed unless it does not precede the escape character. So (1) means an escape char that is floating around not in front of any delimiter is removed. (2) means an escape-escape floating around not in front of any escape char, is preserved. That still doesn't help with your specific issue. If a delimiter begins with the escapeCharacter, will that delimiter appearing in the data be interpreted as an escape character followed by the 2nd and subsequent characters of the delimiter? Or will the delimiter be recognized? Consider dfdl:separator="/ // ///" with escapeCharacter="/" and escapeEscapeCharacter="/" What takes priority, interpretation of escapeCharacters and escapeEscapeCharacters or recognizing delimiters? I have posed this issue for consideration of the other DFDL workgroup members and I'll report back. From: Adams, Joshua Sent: Monday, May 3, 2021 2:38 PM To: dev@daffodil.apache.org Subject: Escape character parsing bug? Consider the following schema: We then have the following test case: foo$$/;bar foo$/;bar Shouldn't this parse as: foo$$ bar The spec says the following: On parsing any in-scope terminating delimiter encountered in the data is not interpreted as such when it is immediately preceded by the dfdl:escapeCharacter (when not itself preceded by the dfdl:escapeEscapeCharacter). Occurrences of the dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are removed from the data, unless the dfdl:escapeCharacter is preceded by the dfdl:escapeEscapeCharacter, or the dfdl:escapeEscapeCharacter does not precede the dfdl:escapeCharacter. It seems to me that the '/;' terminator shouldn't be getting escaped in this case, but want to double check. Josh
