Our team has met with Alex Russell and we are aligned on HEVC support being limited to hardware (no software fallback). With that in mind, I wanted to provide some key conclusions/justifications that will solidify why HEVC is so important to our product and mission.
What is Xbox Cloud Gaming: Microsoft's cloud gaming service that allows users to stream Xbox games directly to their devices over the internet, without requiring a console or high-end PC. Devices include smartphones, tablets, TVs, and web browsers. The game itself runs on a datacenter server and the gameplay (audio/video) is streamed over WebRTC media streams. Why We Need HEVC on WebRTC We want to provide the best gaming experience to our customers by getting the most out of their client endpoint capabilities. - Our cloud gaming scenario requires low latency to ensure a seamless experience for players. Thus, hardware-accelerated encoding and decoding are essential as a baseline requirement (our quality bar isn't met by software encode/decode). - HEVC enables us to provide a high-quality experience not only when streaming to our own hardware (Xbox Consoles) but also when streaming to Smart TVs, Windows, Android, Chrome OS, and Linux endpoints since many of them support HEVC hardware decode. - HEVC allows us to stream at lower bitrates than AVC while maintaining an even higher quality bar and keeping streaming costs lower. - We do want to support AV1 as well, but as Henrik Boström pointed out, the client hardware support for HEVC exceeds that of AV1 right now. - Regarding software vs hardware decode on the client side outside of the HEVC-specific conversation, we understand that MediaCapabilities APIs allow us to query the HW/SW capabilities of the client device, but there is no guarantee that those capabilities will take effect during the actual game stream. Xbox Cloud Gaming and similar scenarios care deeply about what happens during the actual stream (aka we know the device supports HW decode, but was it used?), since we need that data point to make mid-stream decisions (switch codec/resolution/FPS). Thus, we hope that Exposing decode errors / SW fallback as an event · Issue #146 · w3c/webrtc-extensions <https://github.com/w3c/webrtc-extensions/issues/146> is also investigated. Without a reliable way to detect software fallback, our best bet is to monitor decode times (via requestVideoFrameCallback) and have heuristics around what SW and HW decode should look like for a particular device, which is error prone. On Tuesday, March 4, 2025 at 2:34:29 AM UTC-8 Henrik Boström wrote: > AV1 is great from a quality perspective, but it is not an option for use > cases that require hardware acceleration. Diego mentioned potential > server-side requirements, on the client-side estimates are only 8% Windows > endpoints and 0% macOS endpoints have HW accelerated AV1 support. For > mobile clients (including native apps which need to be able to interop with > web clients in video conferencing use cases), it's 0% Android/iOS as well. > As web apps are becoming more taxing on performance and battery (e.g. video > effects processing becoming default) and increased expectations on video > quality, hardware accelerated alternatives is important for both older and > newer devices running modern web apps and for interop with mobile clients. > > H.265 is a different story: HW support is 75% Windows, 99% macOS, 86% > Android, 90% iOS. This goes back to devices from a decade ago which are > still common in the wild. From a quality perspective, this is the only > codec that is close to AV1. H.26*4* does *not* fit that bill. Hardware > acceleration has been measured to reduce power consumption by 20% in basic > calling use cases and above 30% when video effects processing is enabled. > Combine this with the fact that video conferencing use cases draws >10X > more power than when the laptop not doing heavy work (like lite web > browsing) and you can imagine this can have a material impact on your > device's overall battery life, not to mention improving the performance, > unblocking higher resolutions and more CPU budget for features as well as > unblocking HW acceleration in mobile-to-web calling and higher resolution > use cases for the future. > > I do not expect the need for AV1 to go away, but I think a compliment is > needed for the hardware/quality balance. > On Monday, March 3, 2025 at 7:36:40 PM UTC+1 Diego Perez Botero wrote: > >> Alex - AV1 isn't an option for all WebRTC use cases due to hardware >> limitations. For instance, for Xbox Cloud Gaming, we do not have AV1 >> encoding capabilities on our server hardware and would benefit >> substantially from having H265 decode support on browser clients. We can >> already get that working end-to-end with Safari, so Chromium-based browsers >> essentially have a feature gap in their WebRTC implementation at this >> point. Also note that the H265 support being tracked here is only for when >> the client has *hardware *support for the codec, so the licensing >> implications are mitigated. >> >> On Monday, March 3, 2025 at 8:15:16 AM UTC-8 Alex Russell wrote: >> >>> Hey Henrik, >>> >>> I'd like to understand the case for the API OWNERS approving more >>> support for patent encumbered codecs when we have AV1. Apple continues to >>> use its hardware to push closed solutions in this space, and I understand >>> the instinct, but I'd like to see quantified benefits. >>> >>> Best, >>> >>> Alex >>> >>> On Monday, March 3, 2025 at 2:36:23 AM UTC-8 Henrik Boström wrote: >>> >> Contact emails >>>> hb...@chromium.org, jianlin.q...@intel.com, e...@chromium.org >>>> >>>> Specification >>>> WebRTC: https://w3c.github.io/webrtc-pc >>>> >>>> But there are no new API surfaces, just a new mime type addition >>>> exposed in existing APIs - the new mime type ("video/H265") is queryable >>>> via MediaCapabilities API: >>>> >>>> https://w3c.github.io/media-capabilities/#dom-videoconfiguration-contenttype >>>> >>>> If supported it will show up in SDP generated by WebRTC and >>>> sender/receiver capabilities which can be used to set codec preferences >>>> <https://w3c.github.io/webrtc-pc/#dom-rtcrtptransceiver-setcodecpreferences> >>>> (existing >>>> WebRTC API). Note that the codec is only used if successfully negotiated, >>>> meaning if the other endpoint does not support the codec then a different >>>> one is automatically used instead. >>>> >>>> Summary >>>> >>>> This newer codec has increased compression efficiency (higher quality >>>> per bitrate) relative to VP8/VP9/H264 and very strong hardware support >>>> going back over a decade. This translates into improved visual experience, >>>> increased battery life and reduced risk of performance issues. The codec >>>> is >>>> already industry standard and we should support it in WebRTC when provided >>>> by the platform, i.e. if it is available in hardware. This codec is >>>> already >>>> available in WebCodecs (M130) and MediaRecorder APIs (soon? >>>> <https://groups.google.com/u/1/a/chromium.org/g/blink-dev/c/YJ1QijNiHeM>). >>>> Support will be queryable via MediaCapabilities API. Safari has already >>>> shipped support in WebRTC. >>>> >>>> H265 encoding is only available if the user's device and operating >>>> system provide the necessary capabilities as we will not provide a >>>> software >>>> implementation to fall back to. >>>> >>>> Blink component >>>> Blink>WebRTC>Video >>>> <https://issues.chromium.org/issues?q=customfield1222907:%22Blink%3EWebRTC%3EVideo%22> >>>> >>>> TAG review/status >>>> N/A, small incremental change >>>> >>>> Risks >>>> Interoperability and Compatibility >>>> >>>> No significant backwards compat risk since the codec is only used if >>>> both endpoints negotiate it. Interop risk is also very small for the same >>>> reason - note that while this codec is new, needing to negotiate >>>> support else falling back to a different format is not a new dynamic: old >>>> codecs like H264 (that's ...4, not ...5) for example come in different >>>> flavors (profile IDs) where support is also HW and implementation specific >>>> and need to be negotiated, to that regard this is not fundamentally any >>>> different. >>>> >>>> *Gecko*: Standards position link: >>>> https://github.com/mozilla/standards-positions/issues/1188 >>>> (Firefox has some non-WebRTC support for H265 already such as media >>>> playback <https://bugzilla.mozilla.org/show_bug.cgi?id=1924066>). >>>> >>>> *WebKit*: Shipped, including WebRTC support. >>>> >>>> *Web developers*: Strongly positive. This includes both internal and >>>> external developers who have both contributed code and emails asking for >>>> timelines. >>>> >>>> WebView application risks >>>> >>>> None >>>> >>>> Debuggability >>>> >>>> N/A >>>> >>>> Will this feature be supported on all six Blink platforms (Windows, >>>> Mac, Linux, ChromeOS, Android, and Android WebView)? >>>> Yes >>>> >>>> Is this feature fully tested by web-platform-tests >>>> <https://chromium.googlesource.com/chromium/src/+/main/docs/testing/web_platform_tests.md> >>>> ? >>>> Yes >>>> >>>> Flag name on about://flags >>>> No new flags are added specific to this codec but there are existing >>>> flags to disable HW acceleration (any codec) which would also disable >>>> H265. >>>> >>>> Finch feature name >>>> WebRtcAllowH265Send for H265 encoding support and >>>> WebRtcAllowH265Receive for H265 decoding support >>>> >>>> Requires code in //chrome? >>>> False >>>> >>>> Tracking bug >>>> https://crbug.com/391903235 >>>> >>>> Estimated milestones >>>> Shipping on desktop >>>> 136 >>>> Shipping on Android >>>> 136 >>>> Shipping on WebView >>>> 136 >>>> >>>> Anticipated spec changes >>>> >>>> None >>>> >>>> Link to entry on the Chrome Platform Status >>>> https://chromestatus.com/feature/5153479456456704?gate=5188857236291584 >>>> >>> -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscr...@chromium.org. To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/fb442e50-2bf9-4ae5-840d-ae8bafb1f91en%40chromium.org.
