Thanks for the detailed update and your patience Mike. This is definitely not the user experience we're hoping for. Are these regressions that have occurred since the v1.2.0 release?
I'm a big fan of compartmentalization and technologies like docker and python's venv (virtual environment) among many others. Relying on system installs of virtual machines (e.g. Java's JVM) and interpreters (e.g., Python, Perl, etc) introduces uncontrolled variables into our runtime environment, which I believe is one of the major encumbrances that you're experiencing and something I hope we can address as a community. I don't think docker is the right answer here because that introduces another system-level dependency... the docker service. I think packaging a JVM or pulling it in as a VS Code dependency (not a system level dependency like docker) will give us much tighter control over our runtime environment. We'll need to be careful too about how we execute our services on that JVM so its execution environment is properly compartmentalized. In addition to the Daffodil debugger service, the Data Editor part of the extension also uses a service running on the JVM which will benefit from this compartmentalization. This also will allow us to upgrade from Java 8 to something more recent (this will help with domain sockets and named fifos too). Anyway, that's the direction I'd be inclined to head in, but it's definitely worth a discussion and hopefully we can converge on solid design. I had an interesting conversation on this topic with GPT-4 that I think is worth sharing (see: https://chat.openai.com/share/da2ca068-714a-4eec-8a0e-8117069b3125). Each distinct failure that you've encountered in testing RC2 needs to have issues associated with them on the project's GitHub so we can track, triage and prioritize, schedule, and assign them. If you're willing to enter your issues into the GitHub tracker, that would be greatly appreciated and you'll be able to stay in the loop on how those issues progress and help us in the review phase to make sure that each issue is being properly and fully addressed. Shane is out, but I'm sure he'll have some additional follow up and things to try before making new pull requests. On Wed, May 31, 2023 at 7:02 PM Mike Beckerle <mbecke...@apache.org> wrote: > > So, I have the 1.3.0 RC2 vscode extension debugger working now for PCAP w/ > ethernetIP and I can set a breakpoint in ethernetIP.dfdl.xsd and I no > longer get an NPE. > > Configuring this is really far too difficult, and I have to have the > schema working first with sbt in order to even hope to configure it. > > But I'll get back to that later. > > More basic issue is that I am unable to debug even dead-simple errors with > it. I clobbered the magic number in the first bytes of an icmp1.cap file. > This fails an assert at line 97 of pcap.dfdl.xsd, which is the first > element of simple type. > > I step through with the debugger, and nothing displays *anything* about > the assert failure. Execution ends. A little pop-up dialog says it wrote my > infoset out, which it didn't. There's nothing showing that anything went > wrong. All panels on the left are blank. > > If I modify the schema and make any mistake, then I get a blocking dialog > showing the SDEs. This is basically intolerable, and is why I have to go > back to sbt to debug this. You can't even go through the SDEs as you move > the editor. It's a blocking dialog. > > So both for schema compilation, and runtime errors there simply has to be > a non-modal display of errors, probably on the left with variables, watch, > call-stack, and breakpoints. But a quasi-edit-buffer containing them would > also be fine, or even just output them to the terminal window. > > The rest of this email is about the hoops I had to jump through to get > this to work at all. Arguably, these issues are about working with a > multi-component schema, and we could define our objectives to simpler > schemas than this, but that puts it in the realm of toy-only schemas. The > example is just PCAP + ethernetIP, which is not an extreme example. > > The existing shell environment of the terminal definitely causes problems. > But just exiting that so that the debugger starts a new terminal doesn't > fix it, as that inherits the environment from where vscode was started. It > starts a new login shell unless you take steps to prevent that. > > A big trip-up in usability is that the wizard that creates the > launch.json, doesn't actually let you populate the daffodilDebugClaspath. > You have to edit that by hand, and nothing verifies it, so any typo and you > get tedious failures that are hard to figure out. You just get an SDE - if > you are lucky - via a blocking dialog. > > The wizard should help you put directories and corresponding jars on the > daffodilDebugClasspath, and it should not let you put a non-existing > directory or jar file on it. Ideally, at launch time the > daffodilDebugClasspath should be checked, just in case something on it no > longer exists. > > If the debugger happens to find a file on the daffodilDebugClasspath in a > jar, vs. in a regular directory (maybe you have a typo in the regular > directory path), so it passes that and hits the jar, then setting a > breakpoint causes a NPE. Something requires a "hierarchical URL" which a > jar-file URL is not. This is the bug that really has to be fixed so that > you can set breakpoints regardless of whether the source DFDL schema file > was in a jar or not. > > Note: at Owl we have DFDL schemas where the starting root element of the > schema isn't even part of the schema, it's part of a component schema that > is in a jar on the classpath. This happens when you have an > envelope-payload idiom. The schema that combines the envelope and payload > together contains only the glue part that connects the two together. The > root element is the envelope's root element, and that schema is in a jar > file. > > I did not see any feature in launch.json for specifying the name/namespace > of the root element. There is just the "program" element, which must be to > a file in the current schema. That needs to be able to be specified just as > element name + namespace. > > The jars have to be on the daffodilDebugClasspath because of the plugins > that daffodil must find in them. That's understandable, the problem is > having to also have a regular file location for any schema you want to be > able to put a breakpoint in and it must be before the corresponding jar. > > That all said, here are the scripts and settings I used to get it to work: > > First, I have this script I use to launch vscode in a minimal/empty > environment. This technique is useful to > determine if you are depending on the environment in any way: > > #! /bin/bash > if [ "$1" = "emptyEnv" ] # is it the special first arg > then > shift # discard special first arg, and continue with the rest of the > script > else > CMD=$0 > exec -c $CMD "emptyEnv" $* # exec with an empty environment, special > first arg > fi > # the environment is empty when we get here > # just add env vars we need to get vscode to work > export LANG=en_US.UTF-8 > export HOME=/home/mbeckerle > export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 # java 11 required > export PATH=$JAVA_HOME/bin:$PATH > export DISPLAY=:0 > export DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus # vscode > spits lots of errors without this > exec /usr/bin/code > # end > > Then, within ~/.config/Code/User/settings.json, I had to create these > settings for terminal: > > "terminal.integrated.inheritEnv": false, > "terminal.integrated.profiles.linux": { > "Custom Init": { > "path": "/bin/bash", > "args": [ > "--norc" > ] > } > }, > "terminal.integrated.defaultProfile.linux": "Custom Init" > > That instructs vscode not to create a login shell for the terminal. The > --norc option means it does not run the .bashrc file so that the java > version specified in the launch script sticks. > > I am not sure the inheritEnv setting actually does anything. That alone > sounds like the right thing, but that alone didn't solve the problem. > You still have everything in the login shell environment where vscode was > launched. Hence the need for the environment-emptying launch script. > > With those two things, a terminal created by vscode does not inherit > everything my typical shell has. For example there is no daffodil on the > path, no DAFFODIL_CLASSPATH env var, etc. > > Given the above if I just create a terminal with Terminal > new Terminal: > The environment is relatively simple: > > bash-5.0$ env > COLORTERM=truecolor > TERM_PROGRAM_VERSION=1.78.2 > JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 > NO_AT_BRIDGE=1 > PWD=/home/mbeckerle/dataiti/opensource/DFDLSchemas/PCAP > VSCODE_GIT_ASKPASS_NODE=/usr/share/code/code > HOME=/home/mbeckerle > LANG=en_US.UTF-8 > GIT_ASKPASS=/usr/share/code/resources/app/extensions/git/dist/askpass.sh > CHROME_DESKTOP=code-url-handler.desktop > VSCODE_GIT_ASKPASS_EXTRA_ARGS=--ms-enable-electron-run-as-node > TERM=xterm-256color > VSCODE_GIT_IPC_HANDLE=/tmp/vscode-git-56a269c0f5.sock > DISPLAY=:0 > SHLVL=1 > > VSCODE_GIT_ASKPASS_MAIN=/usr/share/code/resources/app/extensions/git/dist/askpass-main.js > GDK_BACKEND=x11 > > PATH=/usr/lib/jvm/java-11-openjdk-amd64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin > ORIGINAL_XDG_CURRENT_DESKTOP=undefined > DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus > TERM_PROGRAM=vscode > _=/usr/bin/env > > If I run daffodil, then c-C in the terminal it creates, then the > environment has this added environment variable only. > > > DAFFODIL_DEBUG_CLASSPATH=/home/mbeckerle/dataiti/opensource/DFDLSchemas/PCAP/../ethernetIP/src/main/resources:/home/mbeckerle/dataiti/opensource/DFDLSchemas/PCAP/lib_managed/jars/com.owlcyberdefense/dfdl-ethernetip/dfdl-ethernetip-1.2.0.jar > > This works. My launch.json is attached. > > > > > On Fri, May 26, 2023 at 2:06 PM Shane Dell <shaned...@apache.org> wrote: > >> Mike, >> >> The VSCode Extension should be using the daffodil command that is bundled >> with the >> extension. The issue might still be with that shell env variable you have >> for >> DAFFODIL_CLASSPATH, but this might only be an issue with a terminal that >> is already opened >> with the name "daffodil-debugger". If you close that terminal, the next >> you time you run it should >> create a new terminal with the env set properly. If you could try this >> and see if it works how you >> expect, that would help us single out this issue out to an already active >> terminal. >> >> Thank you, >> >> Shane Dell >> >> >> On 2023/05/25 23:59:53 Mike Beckerle wrote: >> > There are so many environment interactions.... >> > >> > So I have discovered that the VSCode extension just seems to run >> whatever >> > daffodil command is on the path. It's not isolated at all from the shell >> > environment. >> > >> > I found this out because I have a customized daffodil shell script that >> > setups a bunch of DAFFODIL_CLASSPATH contents, and I couldn't figure out >> > why those were part of the VSCode environment. >> > >> > Shouldn't vscode run exactly the daffodil bundled with it, unless you >> > explicitly configure it not to? >> > >> > That's why it is finding the ethernetIP.dfdl.xsd file in the jar, not in >> > the file system. Because my daffodil command sets up lots of jars on the >> > DAFFODIL_CLASSPATH. >> > >> > It also depends on Java 11, breaks with Java 17. I also don't think this >> > should be inherited from the environment. >> > >> > I really have a hard time using, or even testing the VSCode extension >> if I >> > have to scrub my shell environment of everything in it in order to make >> the >> > vscode extension work. >> > >> > But it is this kind of problem that caused people to invent docker >> > containers. >> > >> > I am going to make a launcher script for VSCode that empties the >> > environment, then supplies only what is absolutely necessary, with no >> > outside environment assumptions. >> > >> > I will try to do this Friday if possible. >> > >> > >> > >> > On Wed, May 24, 2023 at 2:50 PM Shane Dell <shaned...@apache.org> >> wrote: >> > >> > > Mike, >> > > >> > > I am not sure why you would be seeing this. I am able to set the >> > > breakpoint where you >> > > mentioned without any issues. Can you try removing the daffodil-dap >> > > folder? This should be >> > > somewhere like: >> > > /home/$USER/.local/share/daffodil-dap >> > > The issue I am thinking is possibly a bug with the rc1 debugger that >> may >> > > not be present on rc2. >> > > But if the folder daffodil-dap/daffodil-debugger-3.4.0-1.3.0/ already >> > > exists from rc1 then rc2 will >> > > not remake it. So in order to remake it you have to manually remove >> this >> > > folder. Let me know if >> > > this helps. >> > > >> > > Thank you, >> > > >> > > Shane Dell >> > > >> > > On 2023/05/24 15:15:33 Mike Beckerle wrote: >> > > > I have been able to continue evaluating RC2. >> > > > >> > > > My vote is -1 >> > > > >> > > > My testing steps listed below. I am trying to debug PCAP. I have a >> broken >> > > > icmp packet (bad magic number), and also just step through a >> correct icmp >> > > > packet. >> > > > >> > > > I am trying to debug icmp.badMagicNum.cap data file. >> > > > >> > > > I should get a failure of an assert because the magic number is not >> one >> > > of >> > > > the expected values. >> > > > >> > > > I'm just trying to single-step through it until it hits the error. >> > > > >> > > > Once we get to the MagicNumber element, it doesn't position on the >> assert >> > > > while evaluating that expression or even on the element carrying the >> > > assert. >> > > > It jumps back to the top-level xs:schema position. >> > > > >> > > > I continue to click single step. Watching for the error to be >> displayed >> > > in >> > > > the upper left under Variables > Parse > Diagnostics > Errors which >> is >> > > > where I think it ought to go. >> > > > >> > > > At some point the failure is somehow detected as the debug session >> ends. >> > > A >> > > > dialog offers to open /tmp/infoset.xml for me, which I open and it >> is >> > > empty >> > > > of course. >> > > > >> > > > Nothing displayed the error message from the assert, or even >> indicated >> > > what >> > > > went wrong at all. Not the state display in the upper left, not the >> > > > terminal, nor debug console nor output. >> > > > >> > > > The upper left box with "Variables" is cleared at the end. It should >> > > > certainly not be cleared at the end of a run when final state may >> be of >> > > > interest. At the start of a run, or manually, but not at the end of >> a >> > > run. >> > > > >> > > > Second attempt to use it: >> > > > >> > > > I create a second launch config this time for what should be a >> successful >> > > > parse of icmp1.cap >> > > > >> > > > As I single step, the position of the cursor on the schema is >> jumping >> > > > around all over the place back and forth between a containing >> element and >> > > > the child elements it contains. >> > > > The infoset display does not get updated until long after the >> display has >> > > > moved on and is positioned on the next element. >> > > > >> > > > This jumping around gets completely out of hand once you step into >> the >> > > > complexType "Ethernet" which is in the ethernetIP jar file. >> > > > >> > > > At that point, after each child of Ethernet, it jumps back to the >> > > Ethernet >> > > > element in the pcap.dfdl.xsd file, and then back into the >> > > > ethernetIP.dfdl.xsd for the next child, then back to the >> pcap.dfdl.xsd >> > > > file, etc. >> > > > >> > > > All this jumping around basically makes stepping useless. So I >> decide to >> > > > put a breakpoint on the Version element inside the IPv4Header >> element >> > > > inside ethernetIP.dfdl.xsd. >> > > > >> > > > Just adding the breakpoint causes the debug session to end with an >> NPE in >> > > > the dap setBreakpoint. >> > > > >> > > > ERROR o.a.d.d.d.DAPodil - unhandled error >> > > > java.lang.NullPointerException: null >> > > > at >> > > java.base/sun.nio.fs.UnixPath.normalizeAndCheck(UnixPath.java:75) >> > > > at java.base/sun.nio.fs.UnixPath.<init>(UnixPath.java:69) >> > > > at >> > > > java.base/sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:279) >> > > > at java.base/java.nio.file.Path.of(Path.java:147) >> > > > at java.base/java.nio.file.Paths.get(Paths.java:69) >> > > > at >> > > > >> > > >> org.apache.daffodil.debugger.dap.DAPodil.$anonfun$setBreakpoints$1(DAPodil.scala:267) >> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281) >> > > > at flatMap @ >> > > > >> > > >> org.apache.daffodil.debugger.dap.DAPodil.setBreakpoints(DAPodil.scala:263) >> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163) >> > > > at handleErrorWith @ >> > > > fs2.Compiler$Target.handleErrorWith(Compiler.scala:161) >> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163) >> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281) >> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163) >> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163) >> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163) >> > > > at flatMap @ fs2.Pull$.goEval$1(Pull.scala:1057) >> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281) >> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163) >> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163) >> > > > at flatMap @ >> fs2.Pull$.fs2$Pull$$interruptGuard$1(Pull.scala:929) >> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281) >> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163) >> > > > 2023-05-24 11:09:10,717 [io-compute-1] DEBUG o.a.d.d.d.DAPodil - >> > > whenDone: >> > > > completed >> > > > java.lang.NullPointerException >> > > > at >> > > java.base/sun.nio.fs.UnixPath.normalizeAndCheck(UnixPath.java:75) >> > > > at java.base/sun.nio.fs.UnixPath.<init>(UnixPath.java:69) >> > > > at >> > > > java.base/sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:279) >> > > > at java.base/java.nio.file.Path.of(Path.java:147) >> > > > at java.base/java.nio.file.Paths.get(Paths.java:69) >> > > > at >> > > > >> > > >> org.apache.daffodil.debugger.dap.DAPodil.$anonfun$setBreakpoints$1(DAPodil.scala:267) >> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281) >> > > > at flatMap @ >> > > > >> > > >> org.apache.daffodil.debugger.dap.DAPodil.setBreakpoints(DAPodil.scala:263) >> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163) >> > > > at handleErrorWith @ >> > > > fs2.Compiler$Target.handleErrorWith(Compiler.scala:161) >> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163) >> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281) >> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163) >> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163) >> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163) >> > > > at flatMap @ fs2.Pull$.goEval$1(Pull.scala:1057) >> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281) >> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163) >> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163) >> > > > at flatMap @ >> fs2.Pull$.fs2$Pull$$interruptGuard$1(Pull.scala:929) >> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281) >> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163) >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > On Tue, May 23, 2023 at 5:19 PM Mike Beckerle <mbecke...@apache.org >> > >> > > wrote: >> > > > >> > > > > I really am trying to eval the RC2. >> > > > > >> > > > > Right now my VMWare Linux system has been broken by recent >> updates to >> > > the >> > > > > MS Windows host environment. >> > > > > >> > > > > I don't exactly understand this, but the VSCode front end can no >> longer >> > > > > connect to the server at localhost:4711. I get ECONNREFUSED. This >> > > started >> > > > > when a bunch of MS-Windows updates were done on the host OS. I run >> > > VSCode >> > > > > on the linux client machine, which lives behind a NAT firewall >> created >> > > by >> > > > > VMWare workstation - so I really don't get how these could be >> > > interacting >> > > > > this way, but... there it is. >> > > > > >> > > > > I will switch back to my standalone linux PC if this isn't >> resolved >> > > today. >> > > > > >> > > > > >> > > > > On Tue, May 16, 2023 at 1:24 PM Shane Dell <shaned...@apache.org> >> > > wrote: >> > > > > >> > > > >> Hello all,I'd like to call a vote to release Apache Daffodil VS >> Code >> > > > >> 1.3.0-rc2. >> > > > >> >> > > > >> All distribution packages, including signatures, digests, etc. >> can be >> > > > >> found at: >> > > > >> >> > > >> https://dist.apache.org/repos/dist/dev/daffodil/daffodil-vscode/1.3.0-rc2 >> > > > >> >> > > > >> This release has been signed with PGP key >> > > > >> 86DDE7B41291E380237934F007570D3ADC76D51B, corresponding >> > > > >> to shaned...@apache.org, which is included in the KEYS file >> here: >> > > > >> https://downloads.apache.org/daffodil/KEYS >> > > > >> >> > > > >> The release candidate has been tagged in git with 1.3.0-rc2. The >> > > binaries >> > > > >> >> > > > >> For reference, here is a list of all closed GitHub issues tagged >> with >> > > > >> 1.3.0: >> > > > >> https://github.com/apache/daffodil-vscode/milestone/4?closed=1 >> > > > >> >> > > > >> Please review and vote. The vote will be open for at least 72 >> hours >> > > > >> (Friday, 19 May 2023, 1:30pm EST). >> > > > >> >> > > > >> [ ] +1 approve >> > > > >> [ ] +0 no opinion >> > > > >> [ ] -1 disapprove (and reason why) >> > > > >> >> > > > >> Documentation for 1.3.0 can be found here >> > > > >> Apache Daffodil Extension for Visual Studio Code v1.3.0 Wiki >> > > > >> < >> > > > >> >> > > >> https://github.com/apache/daffodil-vscode/wiki/Apache-Daffodil%E2%84%A2-Extension-for-Visual-Studio-Code:-v1.3.0 >> > > > >> >. >> > > > >> >> > > > >> Please note one of the large features that was focused on in >> 1.3.0 was >> > > > >> improving the data editor. To open the data editor you will need >> to >> > > open >> > > > >> the >> > > > >> command palette using: >> > > > >> >> > > > >> - Ctrl + Shift + P (windows/linux) >> > > > >> - Command + Shift + P (mac) >> > > > >> >> > > > >> Then typing "data.edit", make sure "Daffodil Debug: Data Editor" >> is >> > > > >> selected >> > > > >> and then hitting enter. By default the data editor will run on >> port >> > > 9000 >> > > > >> of >> > > > >> your machine. However, if you want to use a different port follow >> > > steps >> > > > >> here >> > > > >> Data Editor Launch Settings >> > > > >> < >> > > > >> >> > > >> https://github.com/apache/daffodil-vscode/wiki/Apache-Daffodil%E2%84%A2-Extension-for-Visual-Studio-Code:-v1.3.0#data-editor-launch-settings >> > > > >> > >> > > > >> >> > > > >> Thank you, >> > > > >> >> > > > >> - Shane Dell >> > > > >> >> > > > > >> > > > >> > > >> > >> >
