Re: [webkit-dev] Implementing the device element
Hi, On 2011-02-28 19:01, Leandro Graciá Gil wrote: I'm glad we agree on the importance of allowing the user to revoke device access. However we think that this point is important at the design level rather than the implementation level. If access revoking is introduced, then the concept of access lifetime arises and needs to be managed within the code. This access lifetime, let's call it session for instance, is exactly the concept we introduced (and misnamed) as 'connections'. These sessions and their management are the core of our proposed model, causing most of the design differences and problems between our proposals. That's why we would like to get this point clear at the design level before submitting any implementation to be patched later. The concept of access lifetime already exists since a device can be physically disconnected at any time and the device handler APIs will need to handle that the underlying device is unavailable. The main concern I have with your session concept is that it involves the device element. Our approach, which is in line with the discussions about the device element being modeled after input type=file, implements the device element as a device selector and a factory for e.g. Stream objects, which no longer depend on the device element that created them (similar to File objects). As a result, the same device element could be used over and over again to create new Stream objects. I assume that is not possible in your design since a device element enters a state where it manages a session to the previously selected device(s)). Having the device element involved in the session feels strange when it can be removed from the page while the Stream object continues to exist. Similar to how File objects work, we do not see a session being started when you select a device, but rather when the device is used. We see the indicator in the browser chrome as a more important feature, since it is uncertain how common it will be that you trust a page enough to grant it access to you camera and then stop trusting it, but still want to stay on the page instead of simply closing it. I also like your proposal of providing access revocation in the same UI used to notify active device sessions. However I'm not sure about how do you plan to do this using the chrome client without explicitly managing the device access lifetime (sessions) in the common WebCore code. It also sounds to me more like an event that WebCore receives from the UA than a request from WebCore to the chrome client. Could you please elaborate on this? As mentioned above, our view of a session differs from yours. As a result, we will not toggle the indicators in the browser chrome based on the device being in use, but rather on information that a page has been granted access to use a device. To toggle the indicators in the chrome, and possibly offer revoking, we will need to expose WebCore/WebKit APIs to explicitly manage device status and access. Also about your proposal for access revocation, I would like to point that it's unintuitive to have to click on one button inside the page to enable access to camera and to have to click on another button, located in a different place altogether, to end this access. Furthermore, consider mobile browsers where the screen real state is extremely expensive. Where would you place this chrome-based UI? To be realistic, I think we'd have to allow both: let the device renderer have state and act as a toggle, as well as allow the possibility to have the device session ended via some UI element in the browser's chrome. I agree with you that it is not ideal to grant and revoke access from two different places. However, having the revoke button (rendered by the device element) in the page is dangerous since it can be replaced by a button that mimics the device element, but actually does not revoke access. This gives a sense of false security. The user must check the indicator in the chrome to verify that the access actually has been revoked and we cannot rely on the user doing that. The fact that the user anyhow must check (and possibly revoke access) from the chrome makes the posibility to revoke from the page questionable. I agree with your last statement about handling the access revocation in the same way that physically unplugging the device. Since some of our discussion points are about design and implementation details for the Stream API, would you mind to upload a patch with your proposal for it? Not necessarily for review, but for allowing all the people interested in this to review and analyze it. The majority of our Stream API implementation platform/component (GStreamer) dependent code that would not add so much to this discusson. Basically the Stream is a quite thin JavaScript object which references pipelines in the media backend. I think that implementation feedback is exactly what we are getting
Re: [webkit-dev] Implementing the device element
Hi, El 22 de febrero de 2011 12:20, Adam Bergkvist adam.bergkv...@ericsson.comescribió: Hi, On 2011-02-15 17:50, Leandro Graciá Gil wrote: Yes, the platform independent code in the device element will determine if a selection has been made, or altered, in the list of available devices Provided by the platform. Before passing it to the device element, the list can be filtered by the platform to exclude some entries for whatever reason. Selections can also be automatically applied without showing the Device selector based on, e.g., stored preferences. The device element will examine the list for selection changes to determine if a new device handler should be created (and a change event dispatched). Consider the following example: 1. The user clicks the device element button and a list of devices is presented. 2. The user selects a webcam in the list and clicks OK. *A new Stream is created* 3. The user once again clicks the device element button and the list is presented with the webcam selected from before. 4. The user clicks OK. *No new Stream is created* 5. The user once again clicks the device element button and the list is presented with the webcam selected from before. 6. The user selects a microphone in the list and clicks OK. *A new Stream is created* This is the behavior that should be consistent between ports. I agree that this behavior should be consisted between ports. However, we seem to be missing a case here: what happens if the user wants to revoke access to a device in the middle of a streaming session? We need to be able to allow this case for obvious privacy reasons. I agree that this is definitively something we need to consider, but perhaps as a (near) future improvement. In fact, we have previously been discussing how to revoke device access and came to the conclusion that it must be done via the browser chrome since the device element can be removed from the page as soon as access has been granted. A natural choice would be to combine this with the mechanism for indicating that a camera/mic is active (discussed in the initial device thread http://lists.w3.org/Archives/Public/public-device-apis/2009Dec/0149.html). That would however require additions to the WebKit API. We have left this out in the first version, simply because it is not mentioned in the spec, and prioritized to land an initial implemenation that could be used as a basis for improvements. I'm glad we agree on the importance of allowing the user to revoke device access. However we think that this point is important at the design level rather than the implementation level. If access revoking is introduced, then the concept of access lifetime arises and needs to be managed within the code. This access lifetime, let's call it session for instance, is exactly the concept we introduced (and misnamed) as 'connections'. These sessions and their management are the core of our proposed model, causing most of the design differences and problems between our proposals. That's why we would like to get this point clear at the design level before submitting any implementation to be patched later. I also like your proposal of providing access revocation in the same UI used to notify active device sessions. However I'm not sure about how do you plan to do this using the chrome client without explicitly managing the device access lifetime (sessions) in the common WebCore code. It also sounds to me more like an event that WebCore receives from the UA than a request from WebCore to the chrome client. Could you please elaborate on this? Also about your proposal for access revocation, I would like to point that it's unintuitive to have to click on one button inside the page to enable access to camera and to have to click on another button, located in a different place altogether, to end this access. Furthermore, consider mobile browsers where the screen real state is extremely expensive. Where would you place this chrome-based UI? To be realistic, I think we'd have to allow both: let the device renderer have state and act as a toggle, as well as allow the possibility to have the device session ended via some UI element in the browser's chrome. We need a way to let the UA maintain the lifetime of Stream objects. These can be killed by the UA at any point (perhaps due to user action or hardware failure), so we need to specify what happens to these objects in such a case. We also need to design our implementation in a way that can handle this situation. As far as I know, and please correct me if I'm wrong, your proposed design doesn't provide the user a clear way to invalidate a stream if he wants to. The Stream's lifetime is something completely handled by the platform-specific client and with no presence at all in the common
Re: [webkit-dev] Implementing the device element
Hi, On 2011-02-15 17:50, Leandro Graciá Gil wrote: Yes, the platform independent code in the device element will determine if a selection has been made, or altered, in the list of available devices Provided by the platform. Before passing it to the device element, the list can be filtered by the platform to exclude some entries for whatever reason. Selections can also be automatically applied without showing the Device selector based on, e.g., stored preferences. The device element will examine the list for selection changes to determine if a new device handler should be created (and a change event dispatched). Consider the following example: 1. The user clicks the device element button and a list of devices is presented. 2. The user selects a webcam in the list and clicks OK. *A new Stream is created* 3. The user once again clicks the device element button and the list is presented with the webcam selected from before. 4. The user clicks OK. *No new Stream is created* 5. The user once again clicks the device element button and the list is presented with the webcam selected from before. 6. The user selects a microphone in the list and clicks OK. *A new Stream is created* This is the behavior that should be consistent between ports. I agree that this behavior should be consisted between ports. However, we seem to be missing a case here: what happens if the user wants to revoke access to a device in the middle of a streaming session? We need to be able to allow this case for obvious privacy reasons. I agree that this is definitively something we need to consider, but perhaps as a (near) future improvement. In fact, we have previously been discussing how to revoke device access and came to the conclusion that it must be done via the browser chrome since the device element can be removed from the page as soon as access has been granted. A natural choice would be to combine this with the mechanism for indicating that a camera/mic is active (discussed in the initial device thread http://lists.w3.org/Archives/Public/public-device-apis/2009Dec/0149.html). That would however require additions to the WebKit API. We have left this out in the first version, simply because it is not mentioned in the spec, and prioritized to land an initial implemenation that could be used as a basis for improvements. We need a way to let the UA maintain the lifetime of Stream objects. These can be killed by the UA at any point (perhaps due to user action or hardware failure), so we need to specify what happens to these objects in such a case. We also need to design our implementation in a way that can handle this situation. As far as I know, and please correct me if I'm wrong, your proposed design doesn't provide the user a clear way to invalidate a stream if he wants to. The Stream's lifetime is something completely handled by the platform-specific client and with no presence at all in the common WebCore code. We think this should also be an essential part of the common behaviour across platforms. The patch for bug 47264 does not contain anything related to Stream; it is a device element only patch (bug 47265 is for the Stream API). We should not confuse the lifetime of Stream objects (or any other device handler API) with the API's way of handling errors (e.g. that its access to the device has been revoked). Revoking access to a device should probably be handled in the same way (in the device handler API) as physically unplugging the device. It will be up to the device handler API (e.g. Stream) and its potential consumers to handle any errors that may occur. Exclusiveness for certain types of devices being one type of error. Another error occurs when a peripheral device is disconnected while in use. Even if you have established a connection to the device via the device element, handling such an error in the device handler API is still required since it is not possible to revoke a device handler instance. I think this impossibility to revoke a device handler instance is currently the essence of our main divergence point, and something that should be discussed in the specification. Let me clearify what I meant. It is not possible to revoke the actual instance of the device handler. The device handler API would need to handle that it no longer has access to the device it was created to control. Do you see anything in our implementation that could not be easily delegated? As I said before, I would be happy to make any changes that would be required to implement the platform specific parts for Chromium. And we thank you for that, but before fixing any implementation details we think that the problem first needs to be discussed at the specification level. According to Hixie, the specification is in a state where
Re: [webkit-dev] Implementing the device element
Hi, El 10 de febrero de 2011 15:15, Adam Bergkvist adam.bergkv...@ericsson.comescribió: Hi, On 2011-02-09 22:10, Leandro Graciá Gil wrote: So, does that mean that a WebCore platform-independent code is going to determine if a new device handler should be created? I think that some UAs may like to, for example, keep a track of trusted pages for specific types of devices to determine if such a handler should be created. And that is not likely to be implemented in the same way by all the different platforms even if they intend to perform a set of basic common security steps. I completely agree with you that the behaviour should be consistent across platforms, especially the security and privacy aspects, but I don't think that forcing the common code to use available device lists is the way to do it. If we want a consistent behaviour we should ask for it in the specification and leave the implementation specific details open, not the other way around. Yes, the platform independent code in the device element will determine if a selection has been made, or altered, in the list of available devices Provided by the platform. Before passing it to the device element, the list can be filtered by the platform to exclude some entries for whatever reason. Selections can also be automatically applied without showing the Device selector based on, e.g., stored preferences. The device element will examine the list for selection changes to determine if a new device handler should be created (and a change event dispatched). Consider the following example: 1. The user clicks the device element button and a list of devices is presented. 2. The user selects a webcam in the list and clicks OK. *A new Stream is created* 3. The user once again clicks the device element button and the list is presented with the webcam selected from before. 4. The user clicks OK. *No new Stream is created* 5. The user once again clicks the device element button and the list is presented with the webcam selected from before. 6. The user selects a microphone in the list and clicks OK. *A new Stream is created* This is the behavior that should be consistent between ports. I agree that this behavior should be consisted between ports. However, we seem to be missing a case here: what happens if the user wants to revoke access to a device in the middle of a streaming session? We need to be able to allow this case for obvious privacy reasons. We need a way to let the UA maintain the lifetime of Stream objects. These can be killed by the UA at any point (perhaps due to user action or hardware failure), so we need to specify what happens to these objects in such a case. We also need to design our implementation in a way that can handle this situation. As far as I know, and please correct me if I'm wrong, your proposed design doesn't provide the user a clear way to invalidate a stream if he wants to. The Stream's lifetime is something completely handled by the platform-specific client and with no presence at all in the common WebCore code. We think this should also be an essential part of the common behaviour across platforms. Because of this reason we would like to raise this discussion to the specification level in the whatwg list. We think that the expected behaviour for the device element is something that still needs to be discussed before facing any implementation-specific details. One important point that a selection-based design misses is that potentially many devices, if not all, will require exclusive access to them. Let me give a example. We have two different pages with device elements. First, a user selects some device (e.g. a microphone) in the first page, but the page makes no use of it for the moment and hence the real connection is not performed. Minutes later and without closing the first page, the same user selects the same device from before on the second device element (if you have only one microphone and an average user this could be very easy to happen) but this page also makes no use for it at that moment. What happens if for example the first page starts making use of the device and causes the connections on the second page to fail? This is almost for sure not the behaviour that the user is expecting or wanting. It will be up to the device handler API (e.g. Stream) and its potential consumers to handle any errors that may occur. Exclusiveness for certain types of devices being one type of error. Another error occurs when a peripheral device is disconnected while in use. Even if you have established a connection to the device via the device element, handling such an error in the device handler API is still required since it is not possible to revoke a device handler instance. I think this impossibility to revoke a device handler instance is currently the essence of our main divergence point, and something that should be discussed
Re: [webkit-dev] Implementing the device element
Hi, On 2011-02-09 22:10, Leandro Graciá Gil wrote: So, does that mean that a WebCore platform-independent code is going to determine if a new device handler should be created? I think that some UAs may like to, for example, keep a track of trusted pages for specific types of devices to determine if such a handler should be created. And that is not likely to be implemented in the same way by all the different platforms even if they intend to perform a set of basic common security steps. I completely agree with you that the behaviour should be consistent across platforms, especially the security and privacy aspects, but I don't think that forcing the common code to use available device lists is the way to do it. If we want a consistent behaviour we should ask for it in the specification and leave the implementation specific details open, not the other way around. Yes, the platform independent code in the device element will determine if a selection has been made, or altered, in the list of available devices Provided by the platform. Before passing it to the device element, the list can be filtered by the platform to exclude some entries for whatever reason. Selections can also be automatically applied without showing the Device selector based on, e.g., stored preferences. The device element will examine the list for selection changes to determine if a new device handler should be created (and a change event dispatched). Consider the following example: 1. The user clicks the device element button and a list of devices is presented. 2. The user selects a webcam in the list and clicks OK. *A new Stream is created* 3. The user once again clicks the device element button and the list is presented with the webcam selected from before. 4. The user clicks OK. *No new Stream is created* 5. The user once again clicks the device element button and the list is presented with the webcam selected from before. 6. The user selects a microphone in the list and clicks OK. *A new Stream is created* This is the behavior that should be consistent between ports. One important point that a selection-based design misses is that potentially many devices, if not all, will require exclusive access to them. Let me give a example. We have two different pages with device elements. First, a user selects some device (e.g. a microphone) in the first page, but the page makes no use of it for the moment and hence the real connection is not performed. Minutes later and without closing the first page, the same user selects the same device from before on the second device element (if you have only one microphone and an average user this could be very easy to happen) but this page also makes no use for it at that moment. What happens if for example the first page starts making use of the device and causes the connections on the second page to fail? This is almost for sure not the behaviour that the user is expecting or wanting. It will be up to the device handler API (e.g. Stream) and its potential consumers to handle any errors that may occur. Exclusiveness for certain types of devices being one type of error. Another error occurs when a peripheral device is disconnected while in use. Even if you have established a connection to the device via the device element, handling such an error in the device handler API is still required since it is not possible to revoke a device handler instance. If we again take input type=file and the File API as a model, there you select a filename from a list and get a File object that represents the file. The file itself can be removed from disk but the File object still exists. When you then attempt to use a FileReader on the File it will fail. Regarding exclusiveness for Stream objects, we don't have that limitation in our implementation. It is possible to select the same webcam in several device elements. Yes, you're right on this. I pointed out in my last email that I don't think that there is any problem in having some platform specific code in WebCore as long as any platform can choose to use it or to easily delegate this task to its own WebKit implementation. Sorry if my proposed class diagram mentioned communicating with WebKit as it's not necessarily the case. We have already fixed this for the new diagrams. Do you see anything in our implementation that could not be easily delegated? As I said before, I would be happy to make any changes that would be required to implement the platform specific parts for Chromium. BR Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Implementing the device element
Hi, El 7 de febrero de 2011 18:56, Adam Bergkvist adam.bergkv...@ericsson.com escribió: Hi, On 2011-02-04 19:21, Leandro Graciá Gil wrote: This is good news! Especially for the situations where WebCore can't directly access the hardware. One existing case of this we should keep in mind are the sandboxed environments, where both the probing and the connections must be requested to somewhere outside the sandbox. Usually this will require to communicate with another process, and in this case asynchronous messages are preferred to avoid delays and to make inter-process communication simpler. However I have to disagree in one point. The specification doesn't say anywhere that we should always present a dialog, only that the device element represents a device selector. The UI has been a central part of the discussions that lead up to the device element proposal. The reason for that is that it should enforce security. See the thread UI for enabling webcam use from untrusted content: http://lists.w3.org/Archives/Public/public-device-apis/2009Dec/0149.html especially http://lists.w3.org/Archives/Public/public-device-apis/2009Dec/0194.html You're right that we shouldn't limit ourselves by referring to the device selector as a dialog. I've renamed ChromeClient::runDeviceDialog to runDeviceSelector and the corresponding client interface accordingly. Thanks. I also think that this name, and more importantly this point of view, is more appropriate. Consider this completely valid use case: instead of asking repeatedly the user to select a device, a UA might decide to create some kind of internal device settings configuration panel to select a set of default devices. Later when visiting a page and clicking the device element the UA will automatically use the preferred devices from its internal settings if they are available and the page is trusted. Where is the dialog here? Couldn't you just see the internal device settings configuration panel, you mention, as the device selector that produces a device list that's reused several times? In that case you would skip the dialog and simply apply the predefined selections (similar to the case where a remember my choices check box would be available in the device dialog). Yes, of course. The reason I proposed that use case was to show that we should not force any dialog, but ask for selection in the way that the UA decides. This was just an example of what I mentioned in the previous paragraph and I'm glad that we agree on this. I agree that the device should perform selection, as the spec says. However as I've already explained I don't think we need for example a selection dialog for all use cases. Considering that we don't explicitly need a dialog to perform the selection, the only reason to bring lists of available devices back to WebCore is to send them again to a client, probably the same one we asked for probing. Also, if we consider the possibility of sandboxed environments then the device connection operation cannot be a synchronous operation as commented before. As mentioned above, I see a point in sending the list of available devices to WebCore to determine if a new Stream (or other device handler) should be created since this behavior should be consistent across platforms, regardless if the device is of type media or fishtank. So, does that mean that a WebCore platform-independent code is going to determine if a new device handler should be created? I think that some UAs may like to, for example, keep a track of trusted pages for specific types of devices to determine if such a handler should be created. And that is not likely to be implemented in the same way by all the different platforms even if they intend to perform a set of basic common security steps. I completely agree with you that the behaviour should be consistent across platforms, especially the security and privacy aspects, but I don't think that forcing the common code to use available device lists is the way to do it. If we want a consistent behaviour we should ask for it in the specification and leave the implementation specific details open, not the other way around. The device connection operation is not handled by the device element. The device element is used to simply select devices (similar to how you select a file with input type=file and get a File object which is just a handle to the actual file). The connection takes place when you use the device, e.g., when you play a Stream in a video element; and that will happen asynchronously. I think that we're not talking about exactly the same thing when we say connection, and probably I should have specified more. I'm not implying that data should flow from the moment we select the device (and we establish a connection in our model). The data should start flowing asynchronously when there is something to consume it, for example a video
Re: [webkit-dev] Implementing the device element
Hi, On 2011-02-04 19:21, Leandro Graciá Gil wrote: This is good news! Especially for the situations where WebCore can't directly access the hardware. One existing case of this we should keep in mind are the sandboxed environments, where both the probing and the connections must be requested to somewhere outside the sandbox. Usually this will require to communicate with another process, and in this case asynchronous messages are preferred to avoid delays and to make inter-process communication simpler. However I have to disagree in one point. The specification doesn't say anywhere that we should always present a dialog, only that the device element represents a device selector. The UI has been a central part of the discussions that lead up to the device element proposal. The reason for that is that it should enforce security. See the thread UI for enabling webcam use from untrusted content: http://lists.w3.org/Archives/Public/public-device-apis/2009Dec/0149.html especially http://lists.w3.org/Archives/Public/public-device-apis/2009Dec/0194.html You're right that we shouldn't limit ourselves by referring to the device selector as a dialog. I've renamed ChromeClient::runDeviceDialog to runDeviceSelector and the corresponding client interface accordingly. Consider this completely valid use case: instead of asking repeatedly the user to select a device, a UA might decide to create some kind of internal device settings configuration panel to select a set of default devices. Later when visiting a page and clicking the device element the UA will automatically use the preferred devices from its internal settings if they are available and the page is trusted. Where is the dialog here? Couldn't you just see the internal device settings configuration panel, you mention, as the device selector that produces a device list that's reused several times? In that case you would skip the dialog and simply apply the predefined selections (similar to the case where a remember my choices check box would be available in the device dialog). I agree that the device should perform selection, as the spec says. However as I've already explained I don't think we need for example a selection dialog for all use cases. Considering that we don't explicitly need a dialog to perform the selection, the only reason to bring lists of available devices back to WebCore is to send them again to a client, probably the same one we asked for probing. Also, if we consider the possibility of sandboxed environments then the device connection operation cannot be a synchronous operation as commented before. As mentioned above, I see a point in sending the list of available devices to WebCore to determine if a new Stream (or other device handler) should be created since this behavior should be consistent across platforms, regardless if the device is of type media or fishtank. The device connection operation is not handled by the device element. The device element is used to simply select devices (similar to how you select a file with input type=file and get a File object which is just a handle to the actual file). The connection takes place when you use the device, e.g., when you play a Stream in a video element; and that will happen asynchronously. Reviewing the design with all these factors leaves us the following scheme: - Request device selection asynchronously to the client (not necessarily using a dialog). - Retrieve the available device list. - Forward the list to a client (probably the same that a moment ago probed the devices) to connect them. Do it asynchronously to keep compatibility with sandboxed environments. - Receive the connection request result and some device specific data. So, the available device lists are being sent back to the device element not for making any specific use of them, but for forwarding them to a connection client in an asynchronous model. Wouldn't it be simpler if we refactor the process in this way? - Request device selection asynchronously to a client. - Receive the connection request result and some device specific data. This is exactly what our model proposes. The same goal can be performed by handling connections to devices instead to actual device lists, especially when we're likely to give back the list to the same client that provided it to us. It also avoids any list handling code outside the clients and to implement an intermediate selection/connection element state. To make our proposal clearer, we have uploaded a patch with most of our WebCore implementation. It can be found here: https://bugs.webkit.org/show_bug.cgi?id=53777 This patch it's not intended to be reviewed (it's too big for that) but to serve as an implementation example of our proposed model. Our original plan was to upload it in small, easy to review pieces. This patch would be intended to be the second of them, after introducing
Re: [webkit-dev] Implementing the device element
El 3 de febrero de 2011 18:59, Adam Bergkvist adam.bergkv...@ericsson.com escribió: Hi, Leandro Graciá Gil wrote: The approach proposed in https://bugs.webkit.org/show_bug.cgi?id=47264 brings the device probing and selection to the WebCore level. It does so by first creating a list of available devices using an interface to a platform-dependent client and then it passes this list to a client dialog to perform the actual selection from it. With its current design both operations are performed synchronously, possibly blocking the device element event handler for a long time. I think this is not a suitable design. The device selector dialog was never intended to block the event loop. That was a mistake. In GTK the main loop continues to run when the dialog is shown so although the click handler is blocked the event loop is not blocked. We should not have expected this behavior on other platforms. I have now added a callback from the dialog to the device element when a selection has been made, making it possible to implement a completely asynchronous dialog. The interface towards the platform for getting a list of available devices is called createDeviceList; it makes no assumption about how the probing is done. The only assumption is that when a device selector dialog is about to be presented to the user, a list of available devices is needed. Typically, probing involves asking the OS kernel for a list of available devices and this operation is usually fast. That's why we in our GTK implementation have chosen to simply do the probing synchronously from createDeviceList. I have now moved the createDeviceList call to device dialog code. This makes it possible for platforms that want to do on-demand probing but fear it might block, to call it off the main loop. This is good news! Especially for the situations where WebCore can't directly access the hardware. One existing case of this we should keep in mind are the sandboxed environments, where both the probing and the connections must be requested to somewhere outside the sandbox. Usually this will require to communicate with another process, and in this case asynchronous messages are preferred to avoid delays and to make inter-process communication simpler. However I have to disagree in one point. The specification doesn't say anywhere that we should always present a dialog, only that the device element represents a device selector. Consider this completely valid use case: instead of asking repeatedly the user to select a device, a UA might decide to create some kind of internal device settings configuration panel to select a set of default devices. Later when visiting a page and clicking the device element the UA will automatically use the preferred devices from its internal settings if they are available and the page is trusted. Where is the dialog here? Given the drawbacks of the approach above, I would like to propose an alternative design that solves these issues. First of all, we think that is not necessary at all to bring the device probing, available device lists, device selection or connection to WebCore. The device element isn't really about actual devices, but about connecting to them. We think that it would be the best for all platforms to actually delegate the actual probing, selection and connecting to them in an asynchronous client-based model and hold only connection and handler information in WebCore. The device element is about devices and specifically about selecting devices. device is inspired by input type=file and so is our implementation. When the change event should trigger on the device element, for example, is something that will eventually be specced out and so the code that handles this logic is best put in WebCore in order to preserve consistency between ports. I agree that the device should perform selection, as the spec says. However as I've already explained I don't think we need for example a selection dialog for all use cases. Considering that we don't explicitly need a dialog to perform the selection, the only reason to bring lists of available devices back to WebCore is to send them again to a client, probably the same one we asked for probing. Also, if we consider the possibility of sandboxed environments then the device connection operation cannot be a synchronous operation as commented before. Reviewing the design with all these factors leaves us the following scheme: - Request device selection asynchronously to the client (not necessarily using a dialog). - Retrieve the available device list. - Forward the list to a client (probably the same that a moment ago probed the devices) to connect them. Do it asynchronously to keep compatibility with sandboxed environments. - Receive the connection request result and some device specific data. So, the available device lists are being sent back to the device element not for making any specific use of
Re: [webkit-dev] Implementing the device element
El 4 de febrero de 2011 18:21, Leandro Graciá Gil leandrogra...@chromium.org escribió: I have also created a small diagram to explain how our implementation works: https://docs.google.com/a/google.com/drawings/edit?id=1jSW-6MJd8mp2qPvwnvZnBVzll6UtBz3r1viZgTE4XVAhl=enauthkey=CPLpy5oJ It seems that my previous link to the diagram has permission problems. I've uploaded it as a pdf file in https://bugs.webkit.org/show_bug.cgi?id=53777 . Sorry for the inconveniences. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Implementing the device element
Hi, Leandro Graciá Gil wrote: The approach proposed in https://bugs.webkit.org/show_bug.cgi?id=47264 brings the device probing and selection to the WebCore level. It does so by first creating a list of available devices using an interface to a platform-dependent client and then it passes this list to a client dialog to perform the actual selection from it. With its current design both operations are performed synchronously, possibly blocking the device element event handler for a long time. I think this is not a suitable design. The device selector dialog was never intended to block the event loop. That was a mistake. In GTK the main loop continues to run when the dialog is shown so although the click handler is blocked the event loop is not blocked. We should not have expected this behavior on other platforms. I have now added a callback from the dialog to the device element when a selection has been made, making it possible to implement a completely asynchronous dialog. The interface towards the platform for getting a list of available devices is called createDeviceList; it makes no assumption about how the probing is done. The only assumption is that when a device selector dialog is about to be presented to the user, a list of available devices is needed. Typically, probing involves asking the OS kernel for a list of available devices and this operation is usually fast. That's why we in our GTK implementation have chosen to simply do the probing synchronously from createDeviceList. I have now moved the createDeviceList call to device dialog code. This makes it possible for platforms that want to do on-demand probing but fear it might block, to call it off the main loop. Given the drawbacks of the approach above, I would like to propose an alternative design that solves these issues. First of all, we think that is not necessary at all to bring the device probing, available device lists, device selection or connection to WebCore. The device element isn't really about actual devices, but about connecting to them. We think that it would be the best for all platforms to actually delegate the actual probing, selection and connecting to them in an asynchronous client-based model and hold only connection and handler information in WebCore. The device element is about devices and specifically about selecting devices. device is inspired by input type=file and so is our implementation. When the change event should trigger on the device element, for example, is something that will eventually be specced out and so the code that handles this logic is best put in WebCore in order to preserve consistency between ports. 4. The device client receives the request and, in a non-blocking way, implements the way it likes the device probing, selection, connecting and all the required security operations (the latter also missing in the existing patch). This allows every UA vendor to make their own choices about the UI for device selection. Every vendor should be able to style the device selector as they want, but the user experience should be coherent between platforms since device is in its essence a security feature and user recognition is an important aspect of that. With this approach, not only we avoid to block WebCore with potentially long operations but we provide a great flexibility to all WebKit platforms to implement the nasty device handling details as they want to. This should also potentially increase the amount of code that can be reused by the different platforms, while avoiding any list matching code that can be found in the existing patch. Can you please elaborate on how putting more code in the platforms, instead of WebCore, will increase code reuse? We would gladly make any changes that could make it easier for you to implement the platform specific parts. BR Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Implementing the device element
I have created a patch to introduce compilation guards for the device element implementation. These should be the same independently of the design approach we decide to take while enabling us to propose more specific patches later. Here's the link: https://bugs.webkit.org/show_bug.cgi?id=53595 Thanks, Leandro El 1 de febrero de 2011 01:43, Leandro Graciá Gil leandrogra...@chromium.org escribió: Hi, We have recently noticed a patch to implement the device element in WebKit. Since this is an important new feature, I thought we should have a discussion about the best way to implement it. Here's the actual specification for it: http://www.whatwg.org/specs/web-apps/current-work/complete/commands.html#devices . The approach proposed in https://bugs.webkit.org/show_bug.cgi?id=47264 brings the device probing and selection to the WebCore level. It does so by first creating a list of available devices using an interface to a platform-dependent client and then it passes this list to a client dialog to perform the actual selection from it. With its current design both operations are performed synchronously, possibly blocking the device element event handler for a long time. I think this is not a suitable design. Given the drawbacks of the approach above, I would like to propose an alternative design that solves these issues. First of all, we think that is not necessary at all to bring the device probing, available device lists, device selection or connection to WebCore. The device element isn't really about actual devices, but about connecting to them. We think that it would be the best for all platforms to actually delegate the actual probing, selection and connecting to them in an asynchronous client-based model and hold only connection and handler information in WebCore. Here's an example of the call flow in our proposal: 0. The device element is created in a no connection state and with an empty device connection descriptor. 1. User clicks the device element (a button, for example). 2. The device element changes its internal state from no connection to connecting, and asks to the device controller to connect to a device of its type sending also the document security origin. 3. The device controller, which handles and maps all the device requests of the page, forwards the connection request to the device page client. This client is implemented by the corresponding WebKit platforms. 4. The device client receives the request and, in a non-blocking way, implements the way it likes the device probing, selection, connecting and all the required security operations (the latter also missing in the existing patch). This allows every UA vendor to make their own choices about the UI for device selection. 5. When the platform has finished connecting to a device, it sends back to the device controller a device connection update message. This message comes with a device connection descriptor that contains the actual connection status, error codes/messages, device ids/names if connected and so on. In case of user cancelation, connection or communication error the same message is sent, but with different device connection descriptor values. 6. The device controller forwards the message to the original device element. The original element sets its connection descriptor to the received object and changes its state accordingly. It will change to connected if the connection was successful or to not connected in case of error. It will also fire the appropriate JS events. 7. If the connection was successful, the element will then create valid stream objects from its data attribute. These will not contain any real streaming data but just a url string. All the actual data is internally managed by the platforms, leaving only handler-like objects in WebCore. The url can be provided for the moment with the connection descriptor object, being later replaced by some type-independent object when device types other than media are defined. Requesting to disconnect the device would work much like requesting a connection, but providing the device descriptor object as a parameter instead. With this approach, not only we avoid to block WebCore with potentially long operations but we provide a great flexibility to all WebKit platforms to implement the nasty device handling details as they want to. This should also potentially increase the amount of code that can be reused by the different platforms, while avoiding any list matching code that can be found in the existing patch. I have a patch that implements this design, but I wanted to get some feedback on this list before sending it out for review. Please don't hesitate to make any suggestions that could help to improve this. Thanks, Leandro ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Implementing the device element
I have created a patch to introduce compilation guards for the device element implementation. These should be the same independently of the design approach we decide to take while enabling us to propose more specific patches later. Here's the link: https://bugs.webkit.org/show_bug.cgi?id=53595 Thanks, Leandro PS: sorry if you received this twice. I used the wrong account before and it seems that it wasn't properly reported as a thread reply in the list archive. El 1 de febrero de 2011 01:43, Leandro Graciá Gil leandrogra...@chromium.org escribió: Hi, We have recently noticed a patch to implement the device element in WebKit. Since this is an important new feature, I thought we should have a discussion about the best way to implement it. Here's the actual specification for it: http://www.whatwg.org/specs/web-apps/current-work/complete/commands.html#devices . The approach proposed in https://bugs.webkit.org/show_bug.cgi?id=47264 brings the device probing and selection to the WebCore level. It does so by first creating a list of available devices using an interface to a platform-dependent client and then it passes this list to a client dialog to perform the actual selection from it. With its current design both operations are performed synchronously, possibly blocking the device element event handler for a long time. I think this is not a suitable design. Given the drawbacks of the approach above, I would like to propose an alternative design that solves these issues. First of all, we think that is not necessary at all to bring the device probing, available device lists, device selection or connection to WebCore. The device element isn't really about actual devices, but about connecting to them. We think that it would be the best for all platforms to actually delegate the actual probing, selection and connecting to them in an asynchronous client-based model and hold only connection and handler information in WebCore. Here's an example of the call flow in our proposal: 0. The device element is created in a no connection state and with an empty device connection descriptor. 1. User clicks the device element (a button, for example). 2. The device element changes its internal state from no connection to connecting, and asks to the device controller to connect to a device of its type sending also the document security origin. 3. The device controller, which handles and maps all the device requests of the page, forwards the connection request to the device page client. This client is implemented by the corresponding WebKit platforms. 4. The device client receives the request and, in a non-blocking way, implements the way it likes the device probing, selection, connecting and all the required security operations (the latter also missing in the existing patch). This allows every UA vendor to make their own choices about the UI for device selection. 5. When the platform has finished connecting to a device, it sends back to the device controller a device connection update message. This message comes with a device connection descriptor that contains the actual connection status, error codes/messages, device ids/names if connected and so on. In case of user cancelation, connection or communication error the same message is sent, but with different device connection descriptor values. 6. The device controller forwards the message to the original device element. The original element sets its connection descriptor to the received object and changes its state accordingly. It will change to connected if the connection was successful or to not connected in case of error. It will also fire the appropriate JS events. 7. If the connection was successful, the element will then create valid stream objects from its data attribute. These will not contain any real streaming data but just a url string. All the actual data is internally managed by the platforms, leaving only handler-like objects in WebCore. The url can be provided for the moment with the connection descriptor object, being later replaced by some type-independent object when device types other than media are defined. Requesting to disconnect the device would work much like requesting a connection, but providing the device descriptor object as a parameter instead. With this approach, not only we avoid to block WebCore with potentially long operations but we provide a great flexibility to all WebKit platforms to implement the nasty device handling details as they want to. This should also potentially increase the amount of code that can be reused by the different platforms, while avoiding any list matching code that can be found in the existing patch. I have a patch that implements this design, but I wanted to get some feedback on this list before sending it out for review. Please don't hesitate to make any suggestions that could help to improve this. Thanks, Leandro
