On Tue, Mar 25, 2025 at 3:13 PM Dale Curtis <dalecur...@chromium.org> wrote:
> On Tue, Mar 25, 2025 at 9:44 AM Stephen Chenney <schen...@chromium.org> > wrote: > >> >> >> On Tue, Mar 25, 2025 at 12:10 PM Chromestatus < >> ad...@cr-status.appspotmail.com> wrote: >> >>> Contact emails dalecur...@chromium.org >>> >>> Explainer None >>> >>> Specification https://www.w3.org/TR/webcodecs >>> >>> Summary >>> >>> Introduces "rotation: int" and "flip: bool" values to various video >>> related interfaces in WebCodecs so that developers can work with frame >>> sources that have orientation (E.g., Android cameras, certain media). The >>> VideoFrame interface grows the ability to create VideoFrames with arbitrary >>> rotation and flip as well as accessors for this information on the >>> VideoFrame object. The VideoDecoderConfig object gains rotation and flip >>> fields that are emitted on decoded VideoFrame objects automatically. The >>> VideoEncoder class gains mechanisms for passing rotation and flip >>> information from encode() to the VideoDecoderConfig emitted as part of >>> EncodedVideoChunkMetadata. If encode() is called with frames with different >>> orientations a non-fatal exception will be thrown. configure() may be used >>> to reset the allowed orientation. >>> >> Back in the day I implemented the CSS image-orientation property, which >> is conceptually addressing the same kind of thing in the image case. There >> were privacy concerns about exposing image metadata content through the >> orientation property, leading to the requirement that cross-origin content >> always render using the image's EXIF orientation. This prevents pages from >> applying the image-orientation property, detecting a change in rendering, >> and hence inferring the metadata. >> >> As I understand the spec link and this intent, it seems like existing >> behavior is to always use the encoded video orientation, and this feature >> allows sites to modify the orientation. If that's so, could it expose >> information about users' cameras via video length/width, thus leading to >> additional fingerprinting surface? Did this come up in the spec discussions? >> > > The existing behavior is to ignore orientation entirely, which leads to > incorrect encoding and decoding. There's no new exposure of information in > this case: > > - VideoFrames can't be created from cross-origin objects > <https://www.w3.org/TR/webcodecs/#videoframe-constructors>, so this > can't leak anything about <image> tags or other CanvasImageSource types. > - The camera width, height, and aspectRatio > <https://developer.mozilla.org/en-US/docs/Web/API/MediaTrackSettings> > are already available through the MediaStreamTrack information > (getUserMedia is permission gated). > - Orientation is already indirectly observable per-frame by drawing to > canvas with and without image-orientation: none. > > Thanks. That clarifies things and addresses any concerns I had. > >> Stephen. >> >> >>> >>> >>> Blink component Blink>Media>WebCodecs >>> <https://issues.chromium.org/issues?q=customfield1222907:%22Blink%3EMedia%3EWebCodecs%22> >>> >>> Search tags webcodecs <http:///features#tags:webcodecs>, videoframe >>> <http:///features#tags:videoframe>, orientation >>> <http:///features#tags:orientation>, videodecoder >>> <http:///features#tags:videodecoder>, videoencoder >>> <http:///features#tags:videoencoder> >>> >>> TAG review None >>> >>> TAG review status Not applicable >>> >>> Risks >>> >>> >>> Interoperability and Compatibility >>> >>> Low, feature designed in conjunction with Firefox and the Media WG. >>> >>> >>> *Gecko*: Positive (https://bugzilla.mozilla.org/show_bug.cgi?id=1955904) >>> Spec co-editor, approved all spec changes, implementation planned. >>> >>> *WebKit*: No signal ( >>> https://github.com/WebKit/standards-positions/issues/475) Participated >>> in some of the standards discussions around this feature. >>> >>> *Web developers*: Positive (https://github.com/w3c/webcodecs/issues/351) >>> Active requests from Zoom and others developers (see issue) who are working >>> with cameras with orientation metadata. >>> >>> *Other signals*: >>> >>> Ergonomics >>> >>> No risk. Small incremental change to WebCodecs API. >>> >>> >>> Activation >>> >>> No risk. Small incremental change to WebCodecs API. MDN documentation >>> will be updated. >>> >>> >>> Security >>> >>> No risk. No new control surfaces. Video elements already have this >>> functionality. >>> >>> >>> WebView application risks >>> >>> Does this intent deprecate or change behavior of existing APIs, such >>> that it has potentially high risk for Android WebView-based applications? >>> >>> Some cameras and media will immediately begin exposing orientation >>> information. In uncommon cases this orientation may change. If orientation >>> changes during encoding through VideoEncoder frames with different >>> orientation than the first are currently encoded with incorrect size and >>> orientation. However the operation completes without a hard error. During >>> discussions in the media working group it was decided that these cases >>> should instead throw a non-fatal exception during encode(). There is >>> already precedent for exception handling during encode() so authors should >>> already be handling this case. >>> >>> >>> Debuggability >>> >>> Rotation information shows up in media configs emitted in media dev >>> tools. >>> >>> >>> Will this feature be supported on all six Blink platforms (Windows, Mac, >>> Linux, ChromeOS, Android, and Android WebView)? Yes >>> >>> Mostly just exposes existing under the hood information used by video >>> elements, so is already supported everywhere. >>> >>> >>> Is this feature fully tested by web-platform-tests >>> <https://chromium.googlesource.com/chromium/src/+/main/docs/testing/web_platform_tests.md> >>> ? Yes >>> >>> >>> https://wpt.fyi/results/webcodecs?label=master&label=experimental&aligned&q=orientation >>> >>> >>> Flag name on about://flags WebCodecsOrientation >>> >>> Finch feature name WebCodecsOrientation >>> >>> Requires code in //chrome? False >>> >>> Tracking bug https://crbug.com/40243431 >>> >>> Measurement WebCodecs has use counters, but no specific use counter has >>> been added for the rotation and flip usage. >>> >>> Availability expectation Available on all platforms except for iOS. >>> >>> Adoption expectation Should be adopted by sites consuming camera feeds >>> with WebCodecs. >>> >>> Estimated milestones >>> Shipping on desktop 137 >>> DevTrial on desktop 136 >>> Shipping on Android 137 >>> DevTrial on Android 136 >>> Shipping on WebView 137 >>> >>> Anticipated spec changes >>> >>> Open questions about a feature may be a source of future web compat or >>> interop issues. Please list open issues (e.g. links to known github issues >>> in the project for the feature specification) whose resolution may >>> introduce web compat/interop risk (e.g., changing to naming or structure of >>> the API in a non-backward-compatible way). >>> Spec changes have already been landed after review with Firefox and the >>> Media Working Group: * https://github.com/w3c/webcodecs/pull/884 * >>> https://github.com/w3c/webcodecs/pull/874 * >>> https://github.com/w3c/webcodecs/pull/840 * >>> https://github.com/w3c/webcodecs/pull/559 >>> >>> Link to entry on the Chrome Platform Status >>> https://chromestatus.com/feature/5128855343595520?gate=5187346531876864 >>> >>> This intent message was generated by Chrome Platform Status >>> <https://chromestatus.com>. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "blink-dev" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to blink-dev+unsubscr...@chromium.org. >>> To view this discussion visit >>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/67e2d562.170a0220.e1a1e.07a6.GAE%40google.com >>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/67e2d562.170a0220.e1a1e.07a6.GAE%40google.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscr...@chromium.org. To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAGsbWzQL9F0yDuG76rhdsa%3DS_wu9NTbTn_Qy1h70GpKhmyC3kA%40mail.gmail.com.
