[blink-dev] Intent to Extend Experiment: WebRTC encoded transform - Modify Metadata functions

[Guido Urdaneta]

Contact emails

gui...@chromium.org, he...@google.com, agpa...@chromium.org


Explainer

https://github.com/palak8669/webrtc-encoded-transform/blob/create-encoded-explainer/create-encoded-explainer.md


Specification

https://github.com/w3c/webrtc-encoded-transform/pull/223/files

Original I2E

https://groups.google.com/a/chromium.org/g/blink-dev/c/yqwA-jXv6VY/m/fCkBkclxAAAJ



Summary

Add a feature to the WebRTC Encoded Transform API that allows manipulating
audio and video frame metadata.

Background: A number of use cases have been identified that require the
manipulation of the metadata of WebRTC encoded frames without decoding them
first.

These include:

- Receiving data in encoded form and forwarding it.

- Sending data that has been encoded previously

- Sending data that has been received in encoded form

In particular, we want to support the use case of glitch-free forwarding of
media coming from multiple redundant peer connections that provide the same
media payloads but with different metadata.

Issue links:  

https://github.com/w3c/webrtc-nv-use-cases/issues/77

https://github.com/w3c/webrtc-nv-use-cases/issues/122

https://github.com/w3c/webrtc-encoded-transform/issues/211



Blink component

Blink>WebRTC



TAG review

The original full spec was reviewed by TAG here:
https://github.com/w3ctag/design-reviews/issues/531

No TAG review has been requested for this incremental change, since it is a
small addition and its API surface might change as a result of feedback
from the origin trial and ongoing discussions in the WebRTC working group.


TAG review status

Issues addressed for the original spec. Not yet requested for the
incremental change.


Chromium Trial Name

RTCEncodedFrameSetMetadata


WebFeature UseCounter name

V8RTCEncodedVideoFrame_SetMetadata_Method


Link to origin trial feedback summary

https://docs.google.com/document/d/1oCgXj0bEhJS6WeJ0T-gV5AVhSaEeVixssqm02m4eU9Q/r/0-f4cSv_CIFwIpVPcIKb89TQ/edit?tab=t.0



Risks


Interoperability and Compatibility

Interoperability risk: There is the risk that other browsers will not
implement this feature. We are mitigating this risk by pursuing consensus
in the W3C WebRTC Working Group so that it becomes part of the WebRTC
encoded transform spec supported by all browsers.


Compatibility risk: This is a new feature intended to support a new use
case. It introduces no breaking changes, so we do not expect any
compatibility issues.


Gecko: No signal requested yet. Positive initial feedback during WG
discussions.


WebKit: No signal requested yet. Positive initial feedback during WG
discussions.


Web developers: Positive


Other signals:


Ergonomics

This feature is an extension to WebRTC Encoded Transform, which itself is
an extension to WebRTC/RTCPeerConnection.


Activation

N/A


Security

No new security risks identified.


WebView application risks

None



Reason this experiment is being extended

We have received positive developer feedback and have validated that the
proposal supports the intended use case, but the API requires some changes
in order to achieve consensus in the W3C WebRTC Working Group. This is the
current status:

1. There is consensus to support the use case, which is already part of the
official "WebRTC Extended Use Cases" document from the WG. See requirement
N39 in https://www.w3.org/TR/webrtc-nv-use-cases/#auction

2. There is consensus to develop a spec proposal that includes changes to
the API surface subject to this origin trial. More specifically, the
initial consensus is to remove the setMetadata method in frames (as
originally proposed) and instead introduce a constructor for creating new
frames given an existing frame and new metadata, following a pattern
already established by WebCodecs. In addition, the WG agreed to the
introduction of a class that allows attaching a source of encoded frames to
the sender of an RTCRtpPeerConnection. See
https://github.com/w3c/webrtc-encoded-transform/issues/211#issuecomment-1853419603
and https://www.w3.org/2023/12/12-webrtc-minutes.html#t06. From the WG's
point of view, both changes are seen as parts of the same package (i.e.,
they will be looked at together, not separately).


In this Intent to Extend Experiment, we are requesting to extend the origin
trial for 3 milestones (124 to 126 inclusive). During this period we'll
introduce  the updated frame API (i.e., the constructor) and will
temporarily keep the setMetadata method to allow developers to migrate to
the new API. The encoded frame source will be pursued by a separate intent.


Ongoing technical constraints

None.



Debuggability

N/A



Will this feature be supported on all six Blink platforms (Windows, Mac,
Linux, ChromeOS, Android, and Android WebView)?

Yes


Is this feature fully tested by web-platform-tests

Re: [blink-dev] Intent to Extend Experiment: WebRTC encoded transform - Modify Metadata functions

[Domenic Denicola]

It's exciting to see the Origin Trial system working as intended, giving us
time to get in-the-field feedback but also stay responsive to ongoing
standards discussions about changing the shape of the API.

Recalling the requirements for extending an origin trial
, we need
to see substantial progress demonstrated in all of the following areas:
draft spec, TAG review, signals requests, outreach for feedback from the
spec community, and WPT tests. I think these have mostly been met, with the
exception of TAG review. Do you think it'd be reasonable to start the TAG
review on the constructor proposal now, or do you think there's still a
significant chance it might change due to WG feedback, such that we're
better of continuing to delay the review?

Also, a question on the explainer below.

On Fri, Jan 26, 2024 at 8:27 PM Guido Urdaneta  wrote:

> Contact emails
>
> gui...@chromium.org, he...@google.com, agpa...@chromium.org
>
>
> Explainer
>
>
> https://github.com/palak8669/webrtc-encoded-transform/blob/create-encoded-explainer/create-encoded-explainer.md
>

Is this explainer up to date with the newest specification proposal? The
specification PR you link below introduces RTCEncodedVideoFrame
and RTCEncodedAudioFrame constructors, but the explainer doesn't seem to
reference either of those. Additionally, the explainer seems to talk about
a clone "operator" (is that a way of saying method?), but I don't see that
in the spec PR.


>
> Specification
>
> https://github.com/w3c/webrtc-encoded-transform/pull/223/files
>
> Original I2E
>
>
> https://groups.google.com/a/chromium.org/g/blink-dev/c/yqwA-jXv6VY/m/fCkBkclxAAAJ
>
>
>
> Summary
>
> Add a feature to the WebRTC Encoded Transform API that allows manipulating
> audio and video frame metadata.
>
> Background: A number of use cases have been identified that require the
> manipulation of the metadata of WebRTC encoded frames without decoding them
> first.
>
> These include:
>
> - Receiving data in encoded form and forwarding it.
>
> - Sending data that has been encoded previously
>
> - Sending data that has been received in encoded form
>
> In particular, we want to support the use case of glitch-free forwarding
> of media coming from multiple redundant peer connections that provide the
> same media payloads but with different metadata.
>
> Issue links:  
>
> https://github.com/w3c/webrtc-nv-use-cases/issues/77
>
> https://github.com/w3c/webrtc-nv-use-cases/issues/122
>
> https://github.com/w3c/webrtc-encoded-transform/issues/211
>
>
>
> Blink component
>
> Blink>WebRTC
> 
>
>
> TAG review
>
> The original full spec was reviewed by TAG here:
> https://github.com/w3ctag/design-reviews/issues/531
>
> No TAG review has been requested for this incremental change, since it is
> a small addition and its API surface might change as a result of feedback
> from the origin trial and ongoing discussions in the WebRTC working group.
>
>
> TAG review status
>
> Issues addressed for the original spec. Not yet requested for the
> incremental change.
>
>
> Chromium Trial Name
>
> RTCEncodedFrameSetMetadata
>
>
> WebFeature UseCounter name
>
> V8RTCEncodedVideoFrame_SetMetadata_Method
>
>
> Link to origin trial feedback summary
>
>
> https://docs.google.com/document/d/1oCgXj0bEhJS6WeJ0T-gV5AVhSaEeVixssqm02m4eU9Q/r/0-f4cSv_CIFwIpVPcIKb89TQ/edit?tab=t.0
>
>
>
> Risks
>
>
> Interoperability and Compatibility
>
> Interoperability risk: There is the risk that other browsers will not
> implement this feature. We are mitigating this risk by pursuing consensus
> in the W3C WebRTC Working Group so that it becomes part of the WebRTC
> encoded transform spec supported by all browsers.
>
>
> Compatibility risk: This is a new feature intended to support a new use
> case. It introduces no breaking changes, so we do not expect any
> compatibility issues.
>
>
> Gecko: No signal requested yet. Positive initial feedback during WG
> discussions.
>
>
> WebKit: No signal requested yet. Positive initial feedback during WG
> discussions.
>
>
> Web developers: Positive
>
>
> Other signals:
>
>
> Ergonomics
>
> This feature is an extension to WebRTC Encoded Transform, which itself is
> an extension to WebRTC/RTCPeerConnection.
>
>
> Activation
>
> N/A
>
>
> Security
>
> No new security risks identified.
>
>
> WebView application risks
>
> None
>
>
>
> Reason this experiment is being extended
>
> We have received positive developer feedback and have validated that the
> proposal supports the intended use case, but the API requires some changes
> in order to achieve consensus in the W3C WebRTC Working Group. This is the
> current status:
>
> 1. There is consensus to support the use case, which is already part of
> the official "WebRTC Extended Use Cases" document from the WG. See
> requirement N39 in https://www.w3

Re: [blink-dev] Intent to Extend Experiment: WebRTC encoded transform - Modify Metadata functions

[Guido Urdaneta]

To clarify about TAG reviews, there is no written policy prohibiting TAG
reviews. The policy (or rather the recommended practice) in a case like
this is to wait until there is consensus in the WG (i.e., the proposed
changes have landed in the official spec) before requesting the TAG review.

BTW, can you send the LGTM again? I forgot to cc blink-dev in my initial
reply and the LGTM was sent only to my email address.

On Thu, Feb 1, 2024 at 8:14 AM Domenic Denicola 
wrote:

> Thanks for updating the explainer!
>
> I'm uneasy about this working group policy prohibiting TAG reviews. Can
> you link to where this policy is stated? If it's indeed not possible for
> WebRTC WG members to request TAG review due to Working Group restrictions,
> then that's probably something the API Owners need to discuss more broadly,
> and consider for the list of acceptable exceptions
> 
> .
>
> Nevertheless, given that the design here has changed significantly over
> the course of the Origin Trial, I think it's reasonable to delay TAG review
> until there's been more developer and community feedback on the new API.
> Just be aware that this could be a blocker when it comes time for Intent to
> Ship, and please file for review early enough that we can get a reply
> before then!
>
> So, LGTM to extend.
>
> On Mon, Jan 29, 2024 at 5:40 PM Guido Urdaneta 
> wrote:
>
>>
>>
>> On Mon, Jan 29, 2024 at 8:35 AM Domenic Denicola 
>> wrote:
>>
>>> It's exciting to see the Origin Trial system working as intended, giving
>>> us time to get in-the-field feedback but also stay responsive to ongoing
>>> standards discussions about changing the shape of the API.
>>>
>>> Recalling the requirements for extending an origin trial
>>> , we
>>> need to see substantial progress demonstrated in all of the following
>>> areas: draft spec, TAG review, signals requests, outreach for feedback from
>>> the spec community, and WPT tests. I think these have mostly been met, with
>>> the exception of TAG review. Do you think it'd be reasonable to start the
>>> TAG review on the constructor proposal now, or do you think there's still a
>>> significant chance it might change due to WG feedback, such that we're
>>> better of continuing to delay the review?
>>>
>>> I think it's too early for a TAG review because the proposal hasn't
>> landed in the spec yet and the WebRTC WG policy nowadays is to wait for
>> specs to be complete before requesting a TAG review.
>>
>>
>>
>>> Also, a question on the explainer below.
>>>
>>> On Fri, Jan 26, 2024 at 8:27 PM Guido Urdaneta 
>>> wrote:
>>>
 Contact emails

 gui...@chromium.org, he...@google.com, agpa...@chromium.org


 Explainer


 https://github.com/palak8669/webrtc-encoded-transform/blob/create-encoded-explainer/create-encoded-explainer.md

>>>
>>> Is this explainer up to date with the newest specification proposal? The
>>> specification PR you link below introduces RTCEncodedVideoFrame
>>> and RTCEncodedAudioFrame constructors, but the explainer doesn't seem to
>>> reference either of those.
>>>
>> It is updated now :)
>> For simplicity, the explainer uses only the RTCEncodedVideoFrame
>> constructor.
>>
>>
>>> Additionally, the explainer seems to talk about a clone "operator" (is
>>> that a way of saying method?), but I don't see that in the spec PR.
>>>
>>>
>> That part is now removed because a clone operation is already part of the
>> spec via structuredClone() and is no longer part of this proposal.
>>
>>
>>
>>>
 Specification

 https://github.com/w3c/webrtc-encoded-transform/pull/223/files

 Original I2E


 https://groups.google.com/a/chromium.org/g/blink-dev/c/yqwA-jXv6VY/m/fCkBkclxAAAJ



 Summary

 Add a feature to the WebRTC Encoded Transform API that allows
 manipulating audio and video frame metadata.

 Background: A number of use cases have been identified that require the
 manipulation of the metadata of WebRTC encoded frames without decoding them
 first.

 These include:

 - Receiving data in encoded form and forwarding it.

 - Sending data that has been encoded previously

 - Sending data that has been received in encoded form

 In particular, we want to support the use case of glitch-free
 forwarding of media coming from multiple redundant peer connections that
 provide the same media payloads but with different metadata.

 Issue links:  

 https://github.com/w3c/webrtc-nv-use-cases/issues/77

 https://github.com/w3c/webrtc-nv-use-cases/issues/122

 https://github.com/w3c/webrtc-encoded-transform/issues/211



 Blink component

 Blink>WebRTC
 

Re: [blink-dev] Intent to Extend Experiment: WebRTC encoded transform - Modify Metadata functions

[Domenic Denicola]

On Thu, Feb 1, 2024 at 5:18 PM Guido Urdaneta  wrote:

> To clarify about TAG reviews, there is no written policy prohibiting TAG
> reviews. The policy (or rather the recommended practice) in a case like
> this is to wait until there is consensus in the WG (i.e., the proposed
> changes have landed in the official spec) before requesting the TAG review.
>

I see. To be clear, this is against the recommended practice for Chromium
developers, which is to file for early TAG review as soon as code is landed
behind a flag

.

But as noted, that's only a recommendation, and not a requirement, so as
long as you file for TAG review at least 1 month before the Intent to Ship
,
that shouldn't be a problem.


>
> BTW, can you send the LGTM again? I forgot to cc blink-dev in my initial
> reply and the LGTM was sent only to my email address.
>

LGTM!


> On Thu, Feb 1, 2024 at 8:14 AM Domenic Denicola 
> wrote:
>
>> Thanks for updating the explainer!
>>
>> I'm uneasy about this working group policy prohibiting TAG reviews. Can
>> you link to where this policy is stated? If it's indeed not possible for
>> WebRTC WG members to request TAG review due to Working Group restrictions,
>> then that's probably something the API Owners need to discuss more broadly,
>> and consider for the list of acceptable exceptions
>> 
>> .
>>
>> Nevertheless, given that the design here has changed significantly over
>> the course of the Origin Trial, I think it's reasonable to delay TAG review
>> until there's been more developer and community feedback on the new API.
>> Just be aware that this could be a blocker when it comes time for Intent to
>> Ship, and please file for review early enough that we can get a reply
>> before then!
>>
>> So, LGTM to extend.
>>
>> On Mon, Jan 29, 2024 at 5:40 PM Guido Urdaneta 
>> wrote:
>>
>>>
>>>
>>> On Mon, Jan 29, 2024 at 8:35 AM Domenic Denicola 
>>> wrote:
>>>
 It's exciting to see the Origin Trial system working as intended,
 giving us time to get in-the-field feedback but also stay responsive to
 ongoing standards discussions about changing the shape of the API.

 Recalling the requirements for extending an origin trial
 , we
 need to see substantial progress demonstrated in all of the following
 areas: draft spec, TAG review, signals requests, outreach for feedback from
 the spec community, and WPT tests. I think these have mostly been met, with
 the exception of TAG review. Do you think it'd be reasonable to start the
 TAG review on the constructor proposal now, or do you think there's still a
 significant chance it might change due to WG feedback, such that we're
 better of continuing to delay the review?

 I think it's too early for a TAG review because the proposal hasn't
>>> landed in the spec yet and the WebRTC WG policy nowadays is to wait for
>>> specs to be complete before requesting a TAG review.
>>>
>>>
>>>
 Also, a question on the explainer below.

 On Fri, Jan 26, 2024 at 8:27 PM Guido Urdaneta 
 wrote:

> Contact emails
>
> gui...@chromium.org, he...@google.com, agpa...@chromium.org
>
>
> Explainer
>
>
> https://github.com/palak8669/webrtc-encoded-transform/blob/create-encoded-explainer/create-encoded-explainer.md
>

 Is this explainer up to date with the newest specification proposal? The
 specification PR you link below introduces RTCEncodedVideoFrame
 and RTCEncodedAudioFrame constructors, but the explainer doesn't seem to
 reference either of those.

>>> It is updated now :)
>>> For simplicity, the explainer uses only the RTCEncodedVideoFrame
>>> constructor.
>>>
>>>
 Additionally, the explainer seems to talk about a clone "operator" (is
 that a way of saying method?), but I don't see that in the spec PR.


>>> That part is now removed because a clone operation is already part of
>>> the spec via structuredClone() and is no longer part of this proposal.
>>>
>>>
>>>

> Specification
>
> https://github.com/w3c/webrtc-encoded-transform/pull/223/files
>
> Original I2E
>
>
> https://groups.google.com/a/chromium.org/g/blink-dev/c/yqwA-jXv6VY/m/fCkBkclxAAAJ
>
>
>
> Summa

[blink-dev] Intent to Extend Experiment: WebRTC encoded transform - Modify Metadata functions (setMetadata method)

[Guido Urdaneta]

Contact emails...@chromium.org, gui...@chromium.org, agpa...@chromium.org

Explainer
https://github.com/guidou/webrtc-extensions/blob/main/constructor-explainer.md

Specification
https://w3c.github.io/webrtc-encoded-transform/#dom-rtcencodedvideoframe-constructor

Summary

Allow WebRTC Encoded Transform API to manipulate audio and video frame
metadata. Some WebRTC Encoded Transform use cases involve manipulation of
not only the payload of encoded video / audio frames but also its metadata.
For example, if a peer connection negotiates a custom codec and an encoded
transform is used to implement part or all of the the custom codec and
needs to set the output codec type as part of the metadata of the output
frame. See https://www.w3.org/2024/04/23-webrtc-minutes.html#t01



Blink componentBlink>WebRTC


TAG reviewThe original full spec was reviewed by TAG here:
https://github.com/w3ctag/design-reviews/issues/531
No TAG review requested yet for the setMetadata method (during the Working
Group discussion it was decided to use a constructor, but interest in the
method version was recently revived).


Other use cases:

https://w3c.github.io/webrtc-nv-use-cases/#live-encoded-media

https://w3c.github.io/webrtc-nv-use-cases/#stored-encoded-media

https://w3c.github.io/webrtc-nv-use-cases/#auction

TAG review statusPending

Chromium Trial NameRTCEncodedFrameSetMetadata

Origin Trial documentation link
https://github.com/palak8669/webrtc-encoded-transform/blob/create-encoded-explainer/create-encoded-explainer.md

WebFeature UseCounter nameRTCEncodedFrameSetMetadata

Risks


Interoperability and Compatibility

Interoperability risk: There is always the risk that other browsers will
not implement this feature. This risk is mitigated by alignment across
browser vendors in the W3C WebRTC Working Group around the spec.
Compatibility risk: This is a new feature intended to support new use
cases. It introduces no breaking changes, so we do not expect any
compatibility issues.

*Gecko*: No signal
However, they have shown interest in reviving the discussion for the
setMetadata() method after achieving consensus on the Custom Codec
negotiation API for WebRTC Encoded Transform.

*WebKit*: No signal

*Web developers*: Positive

*Other signals*:

Ergonomics

This feature is an extension to WebRTC Encoded Transform, which itself is
an extension to WebRTC/RTCPeerConnection.


Activation

No significant risks identified.


Security

No new security risks identified.


WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that
it has potentially high risk for Android WebView-based applications?

No


Goals for experimentation

Determine if the proposed API properly supports the intended use case.


Reason this experiment is being extended

There are two reasons to request this extension: 1. This proposal initially
started as a setMetadata() method on encoded frames, but the result of
discussions in the W3C WebRTC Working Group was that introducing a
constructor (instead of a method) was a better fit for the use cases for
which there was consensus in the WG. After a few iterations over the
constructor API shape, the WG achieved consensus recently and we have sent
an Intent to Ship for that. However, the final version of the constructor
only became available in M126 (the last milestone of the Origin Trial) and
we would like to give developers a little more time to migrate to the
shipped version of the API. 2. After achieving consensus on the constructor
with custom metadata, a new use case has been discussed in the WG that has
revived interest in the original setMetadata() proposal. The WG has
achieved consensus on a new API for custom codec negotiation for which
setMetadata() looks like a better fit than the constructor since it doesn't
require copying the payload of the encoded frame. So the WG might achieve
consensus on adding setMetadata() after all. See the resolution of
https://www.w3.org/2024/04/23-webrtc-minutes.html#t01 Since setMetadata()
might be added to the spec in addition to the constructor, we would like to
extend the trial to allow developers to continue experimenting with it.


Ongoing technical constraints

None


Debuggability

N/A


Will this feature be supported on all six Blink platforms (Windows, Mac,
Linux, ChromeOS, Android, and Android WebView)?Yes

Is this feature fully tested by web-platform-tests

?Yes

https://wpt.fyi/results/webrtc-encoded-transform/tentative/RTCEncodedAudioFrame-metadata.https.html?label=master&label=experimental&aligned
https://wpt.fyi/results/webrtc-encoded-transform/tentative/RTCEncodedVideoFrame-metadata.https.html?label=master&label=experimental&aligned


Flag name on chrome://flags

Finch feature nameRTCEncodedFrameSetMetadata

Non-finch justification

Guarded by a Blink Runt

Re: [blink-dev] Intent to Extend Experiment: WebRTC encoded transform - Modify Metadata functions (setMetadata method)

[Yoav Weiss (@Shopify)]

Do I understand correctly that you want to extend the OT for 3 more
milestones, up to 129 (inclusive)?



On Thu, May 2, 2024 at 2:45 PM Guido Urdaneta  wrote:

> Contact emails...@chromium.org, gui...@chromium.org, agpa...@chromium.org
>
> Explainer
> https://github.com/guidou/webrtc-extensions/blob/main/constructor-explainer.md
>
> Specification
> https://w3c.github.io/webrtc-encoded-transform/#dom-rtcencodedvideoframe-constructor
>
> Summary
>
> Allow WebRTC Encoded Transform API to manipulate audio and video frame
> metadata. Some WebRTC Encoded Transform use cases involve manipulation of
> not only the payload of encoded video / audio frames but also its metadata.
> For example, if a peer connection negotiates a custom codec and an encoded
> transform is used to implement part or all of the the custom codec and
> needs to set the output codec type as part of the metadata of the output
> frame. See https://www.w3.org/2024/04/23-webrtc-minutes.html#t01
>
>
>
> Blink componentBlink>WebRTC
> 
>
> TAG reviewThe original full spec was reviewed by TAG here:
> https://github.com/w3ctag/design-reviews/issues/531
> No TAG review requested yet for the setMetadata method (during the Working
> Group discussion it was decided to use a constructor, but interest in the
> method version was recently revived).
>
>
> Other use cases:
>
> https://w3c.github.io/webrtc-nv-use-cases/#live-encoded-media
>
> https://w3c.github.io/webrtc-nv-use-cases/#stored-encoded-media
>
> https://w3c.github.io/webrtc-nv-use-cases/#auction
>
> TAG review statusPending
>
> Chromium Trial NameRTCEncodedFrameSetMetadata
>
> Origin Trial documentation link
> https://github.com/palak8669/webrtc-encoded-transform/blob/create-encoded-explainer/create-encoded-explainer.md
>
> WebFeature UseCounter nameRTCEncodedFrameSetMetadata
>
> Risks
>
>
> Interoperability and Compatibility
>
> Interoperability risk: There is always the risk that other browsers will
> not implement this feature. This risk is mitigated by alignment across
> browser vendors in the W3C WebRTC Working Group around the spec.
> Compatibility risk: This is a new feature intended to support new use
> cases. It introduces no breaking changes, so we do not expect any
> compatibility issues.
>
> *Gecko*: No signal
> However, they have shown interest in reviving the discussion for the
> setMetadata() method after achieving consensus on the Custom Codec
> negotiation API for WebRTC Encoded Transform.
>
> *WebKit*: No signal
>
> *Web developers*: Positive
>
> *Other signals*:
>
> Ergonomics
>
> This feature is an extension to WebRTC Encoded Transform, which itself is
> an extension to WebRTC/RTCPeerConnection.
>
>
> Activation
>
> No significant risks identified.
>
>
> Security
>
> No new security risks identified.
>
>
> WebView application risks
>
> Does this intent deprecate or change behavior of existing APIs, such that
> it has potentially high risk for Android WebView-based applications?
>
> No
>
>
> Goals for experimentation
>
> Determine if the proposed API properly supports the intended use case.
>
>
> Reason this experiment is being extended
>
> There are two reasons to request this extension: 1. This proposal
> initially started as a setMetadata() method on encoded frames, but the
> result of discussions in the W3C WebRTC Working Group was that introducing
> a constructor (instead of a method) was a better fit for the use cases for
> which there was consensus in the WG. After a few iterations over the
> constructor API shape, the WG achieved consensus recently and we have sent
> an Intent to Ship for that. However, the final version of the constructor
> only became available in M126 (the last milestone of the Origin Trial) and
> we would like to give developers a little more time to migrate to the
> shipped version of the API. 2. After achieving consensus on the constructor
> with custom metadata, a new use case has been discussed in the WG that has
> revived interest in the original setMetadata() proposal. The WG has
> achieved consensus on a new API for custom codec negotiation for which
> setMetadata() looks like a better fit than the constructor since it doesn't
> require copying the payload of the encoded frame. So the WG might achieve
> consensus on adding setMetadata() after all. See the resolution of
> https://www.w3.org/2024/04/23-webrtc-minutes.html#t01 Since setMetadata()
> might be added to the spec in addition to the constructor, we would like to
> extend the trial to allow developers to continue experimenting with it.
>
>
> Ongoing technical constraints
>
> None
>
>
> Debuggability
>
> N/A
>
>
> Will this feature be supported on all six Blink platforms (Windows, Mac,
> Linux, ChromeOS, Android, and Android WebView)?Yes
>
> Is this feature fully tested by web-platform-tests
> 
> ?Yes
>
>
> https://wpt.fyi

Re: [blink-dev] Intent to Extend Experiment: WebRTC encoded transform - Modify Metadata functions (setMetadata method)

[Guido Urdaneta]

Yes, but since the main motivation is to let developers migrate to the
final version of the API (which changed in the last milestone of the
original OT), it would be acceptable to have a shorter extension. This is,
of course, assuming the I2S for the final version of the API is approved.

On Fri, May 3, 2024 at 2:39 PM Yoav Weiss (@Shopify) 
wrote:

> Do I understand correctly that you want to extend the OT for 3 more
> milestones, up to 129 (inclusive)?
>
>
>
> On Thu, May 2, 2024 at 2:45 PM Guido Urdaneta  wrote:
>
>> Contact emails...@chromium.org, gui...@chromium.org, agpa...@chromium.org
>>
>> Explainer
>> https://github.com/guidou/webrtc-extensions/blob/main/constructor-explainer.md
>>
>> Specification
>> https://w3c.github.io/webrtc-encoded-transform/#dom-rtcencodedvideoframe-constructor
>>
>> Summary
>>
>> Allow WebRTC Encoded Transform API to manipulate audio and video frame
>> metadata. Some WebRTC Encoded Transform use cases involve manipulation of
>> not only the payload of encoded video / audio frames but also its metadata.
>> For example, if a peer connection negotiates a custom codec and an encoded
>> transform is used to implement part or all of the the custom codec and
>> needs to set the output codec type as part of the metadata of the output
>> frame. See https://www.w3.org/2024/04/23-webrtc-minutes.html#t01
>>
>>
>>
>> Blink componentBlink>WebRTC
>> 
>>
>> TAG reviewThe original full spec was reviewed by TAG here:
>> https://github.com/w3ctag/design-reviews/issues/531
>> No TAG review requested yet for the setMetadata method (during the
>> Working Group discussion it was decided to use a constructor, but interest
>> in the method version was recently revived).
>>
>>
>> Other use cases:
>>
>> https://w3c.github.io/webrtc-nv-use-cases/#live-encoded-media
>>
>> https://w3c.github.io/webrtc-nv-use-cases/#stored-encoded-media
>>
>> https://w3c.github.io/webrtc-nv-use-cases/#auction
>>
>> TAG review statusPending
>>
>> Chromium Trial NameRTCEncodedFrameSetMetadata
>>
>> Origin Trial documentation link
>> https://github.com/palak8669/webrtc-encoded-transform/blob/create-encoded-explainer/create-encoded-explainer.md
>>
>> WebFeature UseCounter nameRTCEncodedFrameSetMetadata
>>
>> Risks
>>
>>
>> Interoperability and Compatibility
>>
>> Interoperability risk: There is always the risk that other browsers will
>> not implement this feature. This risk is mitigated by alignment across
>> browser vendors in the W3C WebRTC Working Group around the spec.
>> Compatibility risk: This is a new feature intended to support new use
>> cases. It introduces no breaking changes, so we do not expect any
>> compatibility issues.
>>
>> *Gecko*: No signal
>> However, they have shown interest in reviving the discussion for the
>> setMetadata() method after achieving consensus on the Custom Codec
>> negotiation API for WebRTC Encoded Transform.
>>
>> *WebKit*: No signal
>>
>> *Web developers*: Positive
>>
>> *Other signals*:
>>
>> Ergonomics
>>
>> This feature is an extension to WebRTC Encoded Transform, which itself is
>> an extension to WebRTC/RTCPeerConnection.
>>
>>
>> Activation
>>
>> No significant risks identified.
>>
>>
>> Security
>>
>> No new security risks identified.
>>
>>
>> WebView application risks
>>
>> Does this intent deprecate or change behavior of existing APIs, such that
>> it has potentially high risk for Android WebView-based applications?
>>
>> No
>>
>>
>> Goals for experimentation
>>
>> Determine if the proposed API properly supports the intended use case.
>>
>>
>> Reason this experiment is being extended
>>
>> There are two reasons to request this extension: 1. This proposal
>> initially started as a setMetadata() method on encoded frames, but the
>> result of discussions in the W3C WebRTC Working Group was that introducing
>> a constructor (instead of a method) was a better fit for the use cases for
>> which there was consensus in the WG. After a few iterations over the
>> constructor API shape, the WG achieved consensus recently and we have sent
>> an Intent to Ship for that. However, the final version of the constructor
>> only became available in M126 (the last milestone of the Origin Trial) and
>> we would like to give developers a little more time to migrate to the
>> shipped version of the API. 2. After achieving consensus on the constructor
>> with custom metadata, a new use case has been discussed in the WG that has
>> revived interest in the original setMetadata() proposal. The WG has
>> achieved consensus on a new API for custom codec negotiation for which
>> setMetadata() looks like a better fit than the constructor since it doesn't
>> require copying the payload of the encoded frame. So the WG might achieve
>> consensus on adding setMetadata() after all. See the resolution of
>> https://www.w3.org/2024/04/23-webrtc-minutes.html#t01 Since
>> setMetadata() might be added to the spec in addition to the cons

Re: [blink-dev] Intent to Extend Experiment: WebRTC encoded transform - Modify Metadata functions (setMetadata method)

[Yoav Weiss (@Shopify)]

LGTM to extend experimentation until M129 inclusive

On Fri, May 3, 2024 at 5:05 PM Guido Urdaneta  wrote:

> Yes, but since the main motivation is to let developers migrate to the
> final version of the API (which changed in the last milestone of the
> original OT), it would be acceptable to have a shorter extension. This is,
> of course, assuming the I2S for the final version of the API is approved.
>

That's fine! :)


>
> On Fri, May 3, 2024 at 2:39 PM Yoav Weiss (@Shopify) <
> yoavwe...@chromium.org> wrote:
>
>> Do I understand correctly that you want to extend the OT for 3 more
>> milestones, up to 129 (inclusive)?
>>
>>
>>
>> On Thu, May 2, 2024 at 2:45 PM Guido Urdaneta 
>> wrote:
>>
>>> Contact emails...@chromium.org, gui...@chromium.org,
>>> agpa...@chromium.org
>>>
>>> Explainer
>>> https://github.com/guidou/webrtc-extensions/blob/main/constructor-explainer.md
>>>
>>> Specification
>>> https://w3c.github.io/webrtc-encoded-transform/#dom-rtcencodedvideoframe-constructor
>>>
>>> Summary
>>>
>>> Allow WebRTC Encoded Transform API to manipulate audio and video frame
>>> metadata. Some WebRTC Encoded Transform use cases involve manipulation of
>>> not only the payload of encoded video / audio frames but also its metadata.
>>> For example, if a peer connection negotiates a custom codec and an encoded
>>> transform is used to implement part or all of the the custom codec and
>>> needs to set the output codec type as part of the metadata of the output
>>> frame. See https://www.w3.org/2024/04/23-webrtc-minutes.html#t01
>>>
>>>
>>>
>>> Blink componentBlink>WebRTC
>>> 
>>>
>>> TAG reviewThe original full spec was reviewed by TAG here:
>>> https://github.com/w3ctag/design-reviews/issues/531
>>> No TAG review requested yet for the setMetadata method (during the
>>> Working Group discussion it was decided to use a constructor, but interest
>>> in the method version was recently revived).
>>>
>>>
>>> Other use cases:
>>>
>>> https://w3c.github.io/webrtc-nv-use-cases/#live-encoded-media
>>>
>>> https://w3c.github.io/webrtc-nv-use-cases/#stored-encoded-media
>>>
>>> https://w3c.github.io/webrtc-nv-use-cases/#auction
>>>
>>> TAG review statusPending
>>>
>>> Chromium Trial NameRTCEncodedFrameSetMetadata
>>>
>>> Origin Trial documentation link
>>> https://github.com/palak8669/webrtc-encoded-transform/blob/create-encoded-explainer/create-encoded-explainer.md
>>>
>>> WebFeature UseCounter nameRTCEncodedFrameSetMetadata
>>>
>>> Risks
>>>
>>>
>>> Interoperability and Compatibility
>>>
>>> Interoperability risk: There is always the risk that other browsers will
>>> not implement this feature. This risk is mitigated by alignment across
>>> browser vendors in the W3C WebRTC Working Group around the spec.
>>> Compatibility risk: This is a new feature intended to support new use
>>> cases. It introduces no breaking changes, so we do not expect any
>>> compatibility issues.
>>>
>>> *Gecko*: No signal
>>> However, they have shown interest in reviving the discussion for the
>>> setMetadata() method after achieving consensus on the Custom Codec
>>> negotiation API for WebRTC Encoded Transform.
>>>
>>> *WebKit*: No signal
>>>
>>> *Web developers*: Positive
>>>
>>> *Other signals*:
>>>
>>> Ergonomics
>>>
>>> This feature is an extension to WebRTC Encoded Transform, which itself
>>> is an extension to WebRTC/RTCPeerConnection.
>>>
>>>
>>> Activation
>>>
>>> No significant risks identified.
>>>
>>>
>>> Security
>>>
>>> No new security risks identified.
>>>
>>>
>>> WebView application risks
>>>
>>> Does this intent deprecate or change behavior of existing APIs, such
>>> that it has potentially high risk for Android WebView-based applications?
>>>
>>> No
>>>
>>>
>>> Goals for experimentation
>>>
>>> Determine if the proposed API properly supports the intended use case.
>>>
>>>
>>> Reason this experiment is being extended
>>>
>>> There are two reasons to request this extension: 1. This proposal
>>> initially started as a setMetadata() method on encoded frames, but the
>>> result of discussions in the W3C WebRTC Working Group was that introducing
>>> a constructor (instead of a method) was a better fit for the use cases for
>>> which there was consensus in the WG. After a few iterations over the
>>> constructor API shape, the WG achieved consensus recently and we have sent
>>> an Intent to Ship for that. However, the final version of the constructor
>>> only became available in M126 (the last milestone of the Origin Trial) and
>>> we would like to give developers a little more time to migrate to the
>>> shipped version of the API. 2. After achieving consensus on the constructor
>>> with custom metadata, a new use case has been discussed in the WG that has
>>> revived interest in the original setMetadata() proposal. The WG has
>>> achieved consensus on a new API for custom codec negotiation for which
>>> setMetadata() looks like a better fit than the construct