On Tue, Mar 25, 2025 at 9:44 AM Stephen Chenney <schen...@chromium.org> wrote:
> > > On Tue, Mar 25, 2025 at 12:10 PM Chromestatus < > ad...@cr-status.appspotmail.com> wrote: > >> Contact emails dalecur...@chromium.org >> >> Explainer None >> >> Specification https://www.w3.org/TR/webcodecs >> >> Summary >> >> Introduces "rotation: int" and "flip: bool" values to various video >> related interfaces in WebCodecs so that developers can work with frame >> sources that have orientation (E.g., Android cameras, certain media). The >> VideoFrame interface grows the ability to create VideoFrames with arbitrary >> rotation and flip as well as accessors for this information on the >> VideoFrame object. The VideoDecoderConfig object gains rotation and flip >> fields that are emitted on decoded VideoFrame objects automatically. The >> VideoEncoder class gains mechanisms for passing rotation and flip >> information from encode() to the VideoDecoderConfig emitted as part of >> EncodedVideoChunkMetadata. If encode() is called with frames with different >> orientations a non-fatal exception will be thrown. configure() may be used >> to reset the allowed orientation. >> > Back in the day I implemented the CSS image-orientation property, which is > conceptually addressing the same kind of thing in the image case. There > were privacy concerns about exposing image metadata content through the > orientation property, leading to the requirement that cross-origin content > always render using the image's EXIF orientation. This prevents pages from > applying the image-orientation property, detecting a change in rendering, > and hence inferring the metadata. > > As I understand the spec link and this intent, it seems like existing > behavior is to always use the encoded video orientation, and this feature > allows sites to modify the orientation. If that's so, could it expose > information about users' cameras via video length/width, thus leading to > additional fingerprinting surface? Did this come up in the spec discussions? > The existing behavior is to ignore orientation entirely, which leads to incorrect encoding and decoding. There's no new exposure of information in this case: - VideoFrames can't be created from cross-origin objects <https://www.w3.org/TR/webcodecs/#videoframe-constructors>, so this can't leak anything about <image> tags or other CanvasImageSource types. - The camera width, height, and aspectRatio <https://developer.mozilla.org/en-US/docs/Web/API/MediaTrackSettings> are already available through the MediaStreamTrack information (getUserMedia is permission gated). - Orientation is already indirectly observable per-frame by drawing to canvas with and without image-orientation: none. > Stephen. > > >> >> >> Blink component Blink>Media>WebCodecs >> <https://issues.chromium.org/issues?q=customfield1222907:%22Blink%3EMedia%3EWebCodecs%22> >> >> Search tags webcodecs <http:///features#tags:webcodecs>, videoframe >> <http:///features#tags:videoframe>, orientation >> <http:///features#tags:orientation>, videodecoder >> <http:///features#tags:videodecoder>, videoencoder >> <http:///features#tags:videoencoder> >> >> TAG review None >> >> TAG review status Not applicable >> >> Risks >> >> >> Interoperability and Compatibility >> >> Low, feature designed in conjunction with Firefox and the Media WG. >> >> >> *Gecko*: Positive (https://bugzilla.mozilla.org/show_bug.cgi?id=1955904) >> Spec co-editor, approved all spec changes, implementation planned. >> >> *WebKit*: No signal ( >> https://github.com/WebKit/standards-positions/issues/475) Participated >> in some of the standards discussions around this feature. >> >> *Web developers*: Positive (https://github.com/w3c/webcodecs/issues/351) >> Active requests from Zoom and others developers (see issue) who are working >> with cameras with orientation metadata. >> >> *Other signals*: >> >> Ergonomics >> >> No risk. Small incremental change to WebCodecs API. >> >> >> Activation >> >> No risk. Small incremental change to WebCodecs API. MDN documentation >> will be updated. >> >> >> Security >> >> No risk. No new control surfaces. Video elements already have this >> functionality. >> >> >> WebView application risks >> >> Does this intent deprecate or change behavior of existing APIs, such that >> it has potentially high risk for Android WebView-based applications? >> >> Some cameras and media will immediately begin exposing orientation >> information. In uncommon cases this orientation may change. If orientation >> changes during encoding through VideoEncoder frames with different >> orientation than the first are currently encoded with incorrect size and >> orientation. However the operation completes without a hard error. During >> discussions in the media working group it was decided that these cases >> should instead throw a non-fatal exception during encode(). There is >> already precedent for exception handling during encode() so authors should >> already be handling this case. >> >> >> Debuggability >> >> Rotation information shows up in media configs emitted in media dev tools. >> >> >> Will this feature be supported on all six Blink platforms (Windows, Mac, >> Linux, ChromeOS, Android, and Android WebView)? Yes >> >> Mostly just exposes existing under the hood information used by video >> elements, so is already supported everywhere. >> >> >> Is this feature fully tested by web-platform-tests >> <https://chromium.googlesource.com/chromium/src/+/main/docs/testing/web_platform_tests.md> >> ? Yes >> >> >> https://wpt.fyi/results/webcodecs?label=master&label=experimental&aligned&q=orientation >> >> >> Flag name on about://flags WebCodecsOrientation >> >> Finch feature name WebCodecsOrientation >> >> Requires code in //chrome? False >> >> Tracking bug https://crbug.com/40243431 >> >> Measurement WebCodecs has use counters, but no specific use counter has >> been added for the rotation and flip usage. >> >> Availability expectation Available on all platforms except for iOS. >> >> Adoption expectation Should be adopted by sites consuming camera feeds >> with WebCodecs. >> >> Estimated milestones >> Shipping on desktop 137 >> DevTrial on desktop 136 >> Shipping on Android 137 >> DevTrial on Android 136 >> Shipping on WebView 137 >> >> Anticipated spec changes >> >> Open questions about a feature may be a source of future web compat or >> interop issues. Please list open issues (e.g. links to known github issues >> in the project for the feature specification) whose resolution may >> introduce web compat/interop risk (e.g., changing to naming or structure of >> the API in a non-backward-compatible way). >> Spec changes have already been landed after review with Firefox and the >> Media Working Group: * https://github.com/w3c/webcodecs/pull/884 * >> https://github.com/w3c/webcodecs/pull/874 * >> https://github.com/w3c/webcodecs/pull/840 * >> https://github.com/w3c/webcodecs/pull/559 >> >> Link to entry on the Chrome Platform Status >> https://chromestatus.com/feature/5128855343595520?gate=5187346531876864 >> >> This intent message was generated by Chrome Platform Status >> <https://chromestatus.com>. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "blink-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to blink-dev+unsubscr...@chromium.org. >> To view this discussion visit >> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/67e2d562.170a0220.e1a1e.07a6.GAE%40google.com >> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/67e2d562.170a0220.e1a1e.07a6.GAE%40google.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscr...@chromium.org. To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPUDrwf3LSD3PVNLgpFCBwwHqF8tG81Tcn9DY1tyoCvfEnhz%2Bg%40mail.gmail.com.
