Hi Josh, Thanks for taking a look at this and noticing it's kind of fragile and broken :) The idlharness.js setup has evolved step by step from IDL manually copied from specs and inlined into specs, to today's setup of the interfaces/*.idl being updated by automated PRs, but *not* reviewed/merged automatically. (The latest sync is block <https://github.com/web-platform-tests/wpt/pull/38345>, as is common.)
We have all the information (from @webref/idl) we need to generate tests for 99% of what matters, and could do that automatically. However, the current setup needs regular intervention because two things are maintained manually. The first is the dependencies between specs, where foo.idl depends on bar.idl, which could be determined automatically but isn't. The second is the instances of interfaces, which is the idl_array.add_object() <https://github.com/web-platform-tests/wpt/blob/9da79c027d26ac774405ad88a5ecbb11aa6f05c7/permissions/idlharness.any.js#L22-L25> calls and all of the required setup. This could be improved by splitting the automatically generated tests from the bits that need manual setup. In order to not require adding even a boilerplate idlharness.any.js, I think those tests would have to be generated by a build step. Best regards, Philip On Wed, Feb 15, 2023 at 2:29 AM Joshua Bell <jsb...@chromium.org> wrote: > I dug in a bit. Looks like the webgl1 and webgl2 IDLs are indeed extracted > by the workflow and placed in > https://github.com/web-platform-tests/wpt/tree/master/interfaces (yay!) > and the WebXR tests even reference the webgl1 IDLs but there's no > idlharness test that actually exercises them. (boo!) > > A pretty minimal idlharness test should (...) automagically verify > exposure of interfaces in the appropriate global contexts (e.g. you'd have > a window.js and worker.js version); that'd be good to add for WebGL - it > would have caught this issue. Slightly more elaborate tests mint various > objects so the properties can be inspected. You can find examples looking > for files named "idlharness.https.window.js" etc in the various Web > Platform Test (WPT) dirs: https://github.com/web-platform-tests/wpt > > Attention blink-dev lurkers! Adding WPTs is a great way to get started > contributing to the web platform, if anyone out there is looking to help > out! Plenty of mentors around here to help folks out! Ken and I can > probably review. > > On Mon, Feb 13, 2023 at 9:00 PM Ken Russell <k...@chromium.org> wrote: > >> Sorry, not sure. The living WebGL specs are hosted at: >> >> https://registry.khronos.org/webgl/specs/latest/1.0/ >> https://registry.khronos.org/webgl/specs/latest/2.0/ >> >> The IDL is actually auto-extracted from the spec so should be easy to >> ingest and validate against Blink's. >> >> One other thing to consider is that Blink's IDL might not be exactly >> identical to the spec's - for example, Blink uses several extended and >> non-standard Web IDL attributes to optimize the calling convention of >> certain entry points. I also vaguely recall some issues with overloads that >> were legal in Web IDL but needed some hackery to build properly in Blink, >> but not sure about this. >> >> > Re: Blink IDL - the good news is that the idlharness WPTs don't look at > Blink's IDL at all - they consume the spec's IDL and try to verify that the > execution environment they're running in is providing objects that behave > as in the IDL, as much as possible. It's not perfect - argument and return > types can't be determined, for example - but it can catch many things. > Hackery e.g. using overloads to simulate unions shouldn't be an issue, > either because neither can be tested properly or because if we get the > overloads right they're indistinguishable from the unions, as intended. > > But anyway, "simple" things like interface exposure should just work, > since they'll compare the spec IDL and assert that the global contains an > appropriately named property. > > >> -Ken >> >> >> >> On Mon, Feb 13, 2023 at 10:18 AM Joshua Bell <jsb...@chromium.org> wrote: >> >>> Super low priority question: >>> >>> We've got WPTs and tooling that slurps Web IDL from specs and validates >>> it against implementations which in theory should have caught this. This is >>> a fairly fragile process (requires spec to be just right, jobs to be >>> configured, tests to exist, integration bots to successfully run it, >>> failures to get bugs filed, humans to triage, etc) so I'm not surprised >>> this was missed, but do you happen to know where in the pipeline we dropped >>> the ball in this case? >>> >>> >>> >>> >>> On Sun, Feb 12, 2023 at 9:35 AM Kenneth Russell <k...@chromium.org> >>> wrote: >>> >>>> Sorry for the delay replying...this email didn't show up in my Inbox >>>> nor Spam folder. >>>> >>>> The other browsers handle this in the same way. Firefox has supported >>>> WebGL on web workers for some time, and Safari had exactly the same IDL bug >>>> which they just fixed. >>>> >>>> -Ken >>>> >>>> >>>> On Wednesday, February 8, 2023 at 9:35:47 AM UTC-7 slightlyoff via >>>> Chromestatus wrote: >>>> >>>>> LGTM1 Do we know if other browsers that support offscreen canvas >>>>> (Safari TP, e.g.) handle this the same way? Or are we the first to ship? >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "blink-dev" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to blink-dev+unsubscr...@chromium.org. >>>> To view this discussion on the web visit >>>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/1da41865-85f7-449c-89e2-d000bf509d86n%40chromium.org >>>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/1da41865-85f7-449c-89e2-d000bf509d86n%40chromium.org?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- > You received this message because you are subscribed to the Google Groups > "blink-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to blink-dev+unsubscr...@chromium.org. > To view this discussion on the web visit > https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAD649j6E0Zfo4qZ2vg8TGxeqVYMKWwSaqsUwgg6UpbqjA76ZHQ%40mail.gmail.com > <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAD649j6E0Zfo4qZ2vg8TGxeqVYMKWwSaqsUwgg6UpbqjA76ZHQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscr...@chromium.org. To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAARdPYfFBmuYmY%3DOyMsTxyBBMEMvun1FM%3DgV%3De25ryvp1mpmmQ%40mail.gmail.com.
