So, I have the 1.3.0 RC2 vscode extension debugger working now for PCAP w/ ethernetIP and I can set a breakpoint in ethernetIP.dfdl.xsd and I no longer get an NPE.
Configuring this is really far too difficult, and I have to have the schema
working first with sbt in order to even hope to configure it.
But I'll get back to that later.
More basic issue is that I am unable to debug even dead-simple errors with
it. I clobbered the magic number in the first bytes of an icmp1.cap file.
This fails an assert at line 97 of pcap.dfdl.xsd, which is the first
element of simple type.
I step through with the debugger, and nothing displays *anything* about the
assert failure. Execution ends. A little pop-up dialog says it wrote my
infoset out, which it didn't. There's nothing showing that anything went
wrong. All panels on the left are blank.
If I modify the schema and make any mistake, then I get a blocking dialog
showing the SDEs. This is basically intolerable, and is why I have to go
back to sbt to debug this. You can't even go through the SDEs as you move
the editor. It's a blocking dialog.
So both for schema compilation, and runtime errors there simply has to be a
non-modal display of errors, probably on the left with variables, watch,
call-stack, and breakpoints. But a quasi-edit-buffer containing them would
also be fine, or even just output them to the terminal window.
The rest of this email is about the hoops I had to jump through to get this
to work at all. Arguably, these issues are about working with a
multi-component schema, and we could define our objectives to simpler
schemas than this, but that puts it in the realm of toy-only schemas. The
example is just PCAP + ethernetIP, which is not an extreme example.
The existing shell environment of the terminal definitely causes problems.
But just exiting that so that the debugger starts a new terminal doesn't
fix it, as that inherits the environment from where vscode was started. It
starts a new login shell unless you take steps to prevent that.
A big trip-up in usability is that the wizard that creates the launch.json,
doesn't actually let you populate the daffodilDebugClaspath. You have to
edit that by hand, and nothing verifies it, so any typo and you get tedious
failures that are hard to figure out. You just get an SDE - if you are
lucky - via a blocking dialog.
The wizard should help you put directories and corresponding jars on the
daffodilDebugClasspath, and it should not let you put a non-existing
directory or jar file on it. Ideally, at launch time the
daffodilDebugClasspath should be checked, just in case something on it no
longer exists.
If the debugger happens to find a file on the daffodilDebugClasspath in a
jar, vs. in a regular directory (maybe you have a typo in the regular
directory path), so it passes that and hits the jar, then setting a
breakpoint causes a NPE. Something requires a "hierarchical URL" which a
jar-file URL is not. This is the bug that really has to be fixed so that
you can set breakpoints regardless of whether the source DFDL schema file
was in a jar or not.
Note: at Owl we have DFDL schemas where the starting root element of the
schema isn't even part of the schema, it's part of a component schema that
is in a jar on the classpath. This happens when you have an
envelope-payload idiom. The schema that combines the envelope and payload
together contains only the glue part that connects the two together. The
root element is the envelope's root element, and that schema is in a jar
file.
I did not see any feature in launch.json for specifying the name/namespace
of the root element. There is just the "program" element, which must be to
a file in the current schema. That needs to be able to be specified just as
element name + namespace.
The jars have to be on the daffodilDebugClasspath because of the plugins
that daffodil must find in them. That's understandable, the problem is
having to also have a regular file location for any schema you want to be
able to put a breakpoint in and it must be before the corresponding jar.
That all said, here are the scripts and settings I used to get it to work:
First, I have this script I use to launch vscode in a minimal/empty
environment. This technique is useful to
determine if you are depending on the environment in any way:
#! /bin/bash
if [ "$1" = "emptyEnv" ] # is it the special first arg
then
shift # discard special first arg, and continue with the rest of the
script
else
CMD=$0
exec -c $CMD "emptyEnv" $* # exec with an empty environment, special
first arg
fi
# the environment is empty when we get here
# just add env vars we need to get vscode to work
export LANG=en_US.UTF-8
export HOME=/home/mbeckerle
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 # java 11 required
export PATH=$JAVA_HOME/bin:$PATH
export DISPLAY=:0
export DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus # vscode spits
lots of errors without this
exec /usr/bin/code
# end
Then, within ~/.config/Code/User/settings.json, I had to create these
settings for terminal:
"terminal.integrated.inheritEnv": false,
"terminal.integrated.profiles.linux": {
"Custom Init": {
"path": "/bin/bash",
"args": [
"--norc"
]
}
},
"terminal.integrated.defaultProfile.linux": "Custom Init"
That instructs vscode not to create a login shell for the terminal. The
--norc option means it does not run the .bashrc file so that the java
version specified in the launch script sticks.
I am not sure the inheritEnv setting actually does anything. That alone
sounds like the right thing, but that alone didn't solve the problem.
You still have everything in the login shell environment where vscode was
launched. Hence the need for the environment-emptying launch script.
With those two things, a terminal created by vscode does not inherit
everything my typical shell has. For example there is no daffodil on the
path, no DAFFODIL_CLASSPATH env var, etc.
Given the above if I just create a terminal with Terminal > new Terminal:
The environment is relatively simple:
bash-5.0$ env
COLORTERM=truecolor
TERM_PROGRAM_VERSION=1.78.2
JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
NO_AT_BRIDGE=1
PWD=/home/mbeckerle/dataiti/opensource/DFDLSchemas/PCAP
VSCODE_GIT_ASKPASS_NODE=/usr/share/code/code
HOME=/home/mbeckerle
LANG=en_US.UTF-8
GIT_ASKPASS=/usr/share/code/resources/app/extensions/git/dist/askpass.sh
CHROME_DESKTOP=code-url-handler.desktop
VSCODE_GIT_ASKPASS_EXTRA_ARGS=--ms-enable-electron-run-as-node
TERM=xterm-256color
VSCODE_GIT_IPC_HANDLE=/tmp/vscode-git-56a269c0f5.sock
DISPLAY=:0
SHLVL=1
VSCODE_GIT_ASKPASS_MAIN=/usr/share/code/resources/app/extensions/git/dist/askpass-main.js
GDK_BACKEND=x11
PATH=/usr/lib/jvm/java-11-openjdk-amd64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ORIGINAL_XDG_CURRENT_DESKTOP=undefined
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
TERM_PROGRAM=vscode
_=/usr/bin/env
If I run daffodil, then c-C in the terminal it creates, then the
environment has this added environment variable only.
DAFFODIL_DEBUG_CLASSPATH=/home/mbeckerle/dataiti/opensource/DFDLSchemas/PCAP/../ethernetIP/src/main/resources:/home/mbeckerle/dataiti/opensource/DFDLSchemas/PCAP/lib_managed/jars/com.owlcyberdefense/dfdl-ethernetip/dfdl-ethernetip-1.2.0.jar
This works. My launch.json is attached.
On Fri, May 26, 2023 at 2:06 PM Shane Dell <shaned...@apache.org> wrote:
> Mike,
>
> The VSCode Extension should be using the daffodil command that is bundled
> with the
> extension. The issue might still be with that shell env variable you have
> for
> DAFFODIL_CLASSPATH, but this might only be an issue with a terminal that
> is already opened
> with the name "daffodil-debugger". If you close that terminal, the next
> you time you run it should
> create a new terminal with the env set properly. If you could try this and
> see if it works how you
> expect, that would help us single out this issue out to an already active
> terminal.
>
> Thank you,
>
> Shane Dell
>
>
> On 2023/05/25 23:59:53 Mike Beckerle wrote:
> > There are so many environment interactions....
> >
> > So I have discovered that the VSCode extension just seems to run whatever
> > daffodil command is on the path. It's not isolated at all from the shell
> > environment.
> >
> > I found this out because I have a customized daffodil shell script that
> > setups a bunch of DAFFODIL_CLASSPATH contents, and I couldn't figure out
> > why those were part of the VSCode environment.
> >
> > Shouldn't vscode run exactly the daffodil bundled with it, unless you
> > explicitly configure it not to?
> >
> > That's why it is finding the ethernetIP.dfdl.xsd file in the jar, not in
> > the file system. Because my daffodil command sets up lots of jars on the
> > DAFFODIL_CLASSPATH.
> >
> > It also depends on Java 11, breaks with Java 17. I also don't think this
> > should be inherited from the environment.
> >
> > I really have a hard time using, or even testing the VSCode extension if
> I
> > have to scrub my shell environment of everything in it in order to make
> the
> > vscode extension work.
> >
> > But it is this kind of problem that caused people to invent docker
> > containers.
> >
> > I am going to make a launcher script for VSCode that empties the
> > environment, then supplies only what is absolutely necessary, with no
> > outside environment assumptions.
> >
> > I will try to do this Friday if possible.
> >
> >
> >
> > On Wed, May 24, 2023 at 2:50 PM Shane Dell <shaned...@apache.org> wrote:
> >
> > > Mike,
> > >
> > > I am not sure why you would be seeing this. I am able to set the
> > > breakpoint where you
> > > mentioned without any issues. Can you try removing the daffodil-dap
> > > folder? This should be
> > > somewhere like:
> > > /home/$USER/.local/share/daffodil-dap
> > > The issue I am thinking is possibly a bug with the rc1 debugger that
> may
> > > not be present on rc2.
> > > But if the folder daffodil-dap/daffodil-debugger-3.4.0-1.3.0/ already
> > > exists from rc1 then rc2 will
> > > not remake it. So in order to remake it you have to manually remove
> this
> > > folder. Let me know if
> > > this helps.
> > >
> > > Thank you,
> > >
> > > Shane Dell
> > >
> > > On 2023/05/24 15:15:33 Mike Beckerle wrote:
> > > > I have been able to continue evaluating RC2.
> > > >
> > > > My vote is -1
> > > >
> > > > My testing steps listed below. I am trying to debug PCAP. I have a
> broken
> > > > icmp packet (bad magic number), and also just step through a correct
> icmp
> > > > packet.
> > > >
> > > > I am trying to debug icmp.badMagicNum.cap data file.
> > > >
> > > > I should get a failure of an assert because the magic number is not
> one
> > > of
> > > > the expected values.
> > > >
> > > > I'm just trying to single-step through it until it hits the error.
> > > >
> > > > Once we get to the MagicNumber element, it doesn't position on the
> assert
> > > > while evaluating that expression or even on the element carrying the
> > > assert.
> > > > It jumps back to the top-level xs:schema position.
> > > >
> > > > I continue to click single step. Watching for the error to be
> displayed
> > > in
> > > > the upper left under Variables > Parse > Diagnostics > Errors which
> is
> > > > where I think it ought to go.
> > > >
> > > > At some point the failure is somehow detected as the debug session
> ends.
> > > A
> > > > dialog offers to open /tmp/infoset.xml for me, which I open and it is
> > > empty
> > > > of course.
> > > >
> > > > Nothing displayed the error message from the assert, or even
> indicated
> > > what
> > > > went wrong at all. Not the state display in the upper left, not the
> > > > terminal, nor debug console nor output.
> > > >
> > > > The upper left box with "Variables" is cleared at the end. It should
> > > > certainly not be cleared at the end of a run when final state may be
> of
> > > > interest. At the start of a run, or manually, but not at the end of a
> > > run.
> > > >
> > > > Second attempt to use it:
> > > >
> > > > I create a second launch config this time for what should be a
> successful
> > > > parse of icmp1.cap
> > > >
> > > > As I single step, the position of the cursor on the schema is jumping
> > > > around all over the place back and forth between a containing
> element and
> > > > the child elements it contains.
> > > > The infoset display does not get updated until long after the
> display has
> > > > moved on and is positioned on the next element.
> > > >
> > > > This jumping around gets completely out of hand once you step into
> the
> > > > complexType "Ethernet" which is in the ethernetIP jar file.
> > > >
> > > > At that point, after each child of Ethernet, it jumps back to the
> > > Ethernet
> > > > element in the pcap.dfdl.xsd file, and then back into the
> > > > ethernetIP.dfdl.xsd for the next child, then back to the
> pcap.dfdl.xsd
> > > > file, etc.
> > > >
> > > > All this jumping around basically makes stepping useless. So I
> decide to
> > > > put a breakpoint on the Version element inside the IPv4Header element
> > > > inside ethernetIP.dfdl.xsd.
> > > >
> > > > Just adding the breakpoint causes the debug session to end with an
> NPE in
> > > > the dap setBreakpoint.
> > > >
> > > > ERROR o.a.d.d.d.DAPodil - unhandled error
> > > > java.lang.NullPointerException: null
> > > > at
> > > java.base/sun.nio.fs.UnixPath.normalizeAndCheck(UnixPath.java:75)
> > > > at java.base/sun.nio.fs.UnixPath.<init>(UnixPath.java:69)
> > > > at
> > > > java.base/sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:279)
> > > > at java.base/java.nio.file.Path.of(Path.java:147)
> > > > at java.base/java.nio.file.Paths.get(Paths.java:69)
> > > > at
> > > >
> > >
> org.apache.daffodil.debugger.dap.DAPodil.$anonfun$setBreakpoints$1(DAPodil.scala:267)
> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281)
> > > > at flatMap @
> > > >
> > >
> org.apache.daffodil.debugger.dap.DAPodil.setBreakpoints(DAPodil.scala:263)
> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
> > > > at handleErrorWith @
> > > > fs2.Compiler$Target.handleErrorWith(Compiler.scala:161)
> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281)
> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
> > > > at flatMap @ fs2.Pull$.goEval$1(Pull.scala:1057)
> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281)
> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
> > > > at flatMap @
> fs2.Pull$.fs2$Pull$$interruptGuard$1(Pull.scala:929)
> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281)
> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
> > > > 2023-05-24 11:09:10,717 [io-compute-1] DEBUG o.a.d.d.d.DAPodil -
> > > whenDone:
> > > > completed
> > > > java.lang.NullPointerException
> > > > at
> > > java.base/sun.nio.fs.UnixPath.normalizeAndCheck(UnixPath.java:75)
> > > > at java.base/sun.nio.fs.UnixPath.<init>(UnixPath.java:69)
> > > > at
> > > > java.base/sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:279)
> > > > at java.base/java.nio.file.Path.of(Path.java:147)
> > > > at java.base/java.nio.file.Paths.get(Paths.java:69)
> > > > at
> > > >
> > >
> org.apache.daffodil.debugger.dap.DAPodil.$anonfun$setBreakpoints$1(DAPodil.scala:267)
> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281)
> > > > at flatMap @
> > > >
> > >
> org.apache.daffodil.debugger.dap.DAPodil.setBreakpoints(DAPodil.scala:263)
> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
> > > > at handleErrorWith @
> > > > fs2.Compiler$Target.handleErrorWith(Compiler.scala:161)
> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281)
> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
> > > > at flatMap @ fs2.Pull$.goEval$1(Pull.scala:1057)
> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281)
> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
> > > > at flatMap @
> fs2.Pull$.fs2$Pull$$interruptGuard$1(Pull.scala:929)
> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281)
> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, May 23, 2023 at 5:19 PM Mike Beckerle <mbecke...@apache.org>
> > > wrote:
> > > >
> > > > > I really am trying to eval the RC2.
> > > > >
> > > > > Right now my VMWare Linux system has been broken by recent updates
> to
> > > the
> > > > > MS Windows host environment.
> > > > >
> > > > > I don't exactly understand this, but the VSCode front end can no
> longer
> > > > > connect to the server at localhost:4711. I get ECONNREFUSED. This
> > > started
> > > > > when a bunch of MS-Windows updates were done on the host OS. I run
> > > VSCode
> > > > > on the linux client machine, which lives behind a NAT firewall
> created
> > > by
> > > > > VMWare workstation - so I really don't get how these could be
> > > interacting
> > > > > this way, but... there it is.
> > > > >
> > > > > I will switch back to my standalone linux PC if this isn't resolved
> > > today.
> > > > >
> > > > >
> > > > > On Tue, May 16, 2023 at 1:24 PM Shane Dell <shaned...@apache.org>
> > > wrote:
> > > > >
> > > > >> Hello all,I'd like to call a vote to release Apache Daffodil VS
> Code
> > > > >> 1.3.0-rc2.
> > > > >>
> > > > >> All distribution packages, including signatures, digests, etc.
> can be
> > > > >> found at:
> > > > >>
> > >
> https://dist.apache.org/repos/dist/dev/daffodil/daffodil-vscode/1.3.0-rc2
> > > > >>
> > > > >> This release has been signed with PGP key
> > > > >> 86DDE7B41291E380237934F007570D3ADC76D51B, corresponding
> > > > >> to shaned...@apache.org, which is included in the KEYS file here:
> > > > >> https://downloads.apache.org/daffodil/KEYS
> > > > >>
> > > > >> The release candidate has been tagged in git with 1.3.0-rc2. The
> > > binaries
> > > > >>
> > > > >> For reference, here is a list of all closed GitHub issues tagged
> with
> > > > >> 1.3.0:
> > > > >> https://github.com/apache/daffodil-vscode/milestone/4?closed=1
> > > > >>
> > > > >> Please review and vote. The vote will be open for at least 72
> hours
> > > > >> (Friday, 19 May 2023, 1:30pm EST).
> > > > >>
> > > > >> [ ] +1 approve
> > > > >> [ ] +0 no opinion
> > > > >> [ ] -1 disapprove (and reason why)
> > > > >>
> > > > >> Documentation for 1.3.0 can be found here
> > > > >> Apache Daffodil Extension for Visual Studio Code v1.3.0 Wiki
> > > > >> <
> > > > >>
> > >
> https://github.com/apache/daffodil-vscode/wiki/Apache-Daffodil%E2%84%A2-Extension-for-Visual-Studio-Code:-v1.3.0
> > > > >> >.
> > > > >>
> > > > >> Please note one of the large features that was focused on in
> 1.3.0 was
> > > > >> improving the data editor. To open the data editor you will need
> to
> > > open
> > > > >> the
> > > > >> command palette using:
> > > > >>
> > > > >> - Ctrl + Shift + P (windows/linux)
> > > > >> - Command + Shift + P (mac)
> > > > >>
> > > > >> Then typing "data.edit", make sure "Daffodil Debug: Data Editor"
> is
> > > > >> selected
> > > > >> and then hitting enter. By default the data editor will run on
> port
> > > 9000
> > > > >> of
> > > > >> your machine. However, if you want to use a different port follow
> > > steps
> > > > >> here
> > > > >> Data Editor Launch Settings
> > > > >> <
> > > > >>
> > >
> https://github.com/apache/daffodil-vscode/wiki/Apache-Daffodil%E2%84%A2-Extension-for-Visual-Studio-Code:-v1.3.0#data-editor-launch-settings
> > > > >> >
> > > > >>
> > > > >> Thank you,
> > > > >>
> > > > >> - Shane Dell
> > > > >>
> > > > >
> > > >
> > >
> >
>
launch.json
Description: application/json
