Bug#863915: unblock: webkit2gtk/2.16.3-2

[Adam D. Barratt]

On Fri, 2017-06-02 at 11:52 -0400, Jeremy Bicha wrote:
> Here's my thinking as to how the first webkit2gtk stable update could work.
> 
> 1. A new webkit2gtk point release is released.
> 2. Since regressions are generally found within the first week and to
> try to limit the work needed by the SRMs, we wait a week before
> uploading to the s-p-u queue.
> 3. A SRM accepts it.
> 4. An email is sent out to the maintainers of the rdeps asking them to
> please test their package with the new webkit2gtk version in s-p-u
> within the next 2 weeks.
> 5. If no regressions are reported, the update is released in the next
> Debian 9 point release.

Has this been discussed with any of the r-dep maintainers? The plan only
really works if step 4 actually results in useful tests.

Regards,

Adam

Bug#863915: unblock: webkit2gtk/2.16.3-2

[Moritz Mühlenhoff]

Adam wrote:

> I'm not entirely sure how you think p-u is better placed to do so, given 
> the amount of visible testing packages from it get before a point 
> release.

It's not necessarily for the additional testing done on p-u (although
I personally use it like that and probably others well), but there's
a number of technical features which make spu "suck less" which are
currently lacking in the security.debian.org infrastructure:
- Lack of visible apt source for people to test (#817286) (biggest blocker)
- Bottleneck of not being able to delegate allowing maintainers of webkit
  rdeps to release compatibility updates via security.debian.org (#817285)
- No possibility to trigger binNMUs of rdeps without a sourceful upload
  (not sure if that's necessary for the changes imposed by newer webkit
  releases, but it's also a serious problem for go-based apps

Especially the first two points are critical to address mid-term if we
want to ensure security support is sustainable in the years to come.
Either by finding new volunteers to work on that or by funding the
development of these features in some way.

Cheers,
Moritz
 

Bug#863915: unblock: webkit2gtk/2.16.3-2

[Jeremy Bicha]

I apologize for not personally trying hard to push this through, but
if we don't unblock 2.16.3 in the next few days, it makes the Stable
Release process more difficult since we can't "practice" with easier
webkit2gtk point releases first.

Proposal for initial webkit2gtk updates in Debian 9

Here's my thinking as to how the first webkit2gtk stable update could work.

1. A new webkit2gtk point release is released.
2. Since regressions are generally found within the first week and to
try to limit the work needed by the SRMs, we wait a week before
uploading to the s-p-u queue.
3. A SRM accepts it.
4. An email is sent out to the maintainers of the rdeps asking them to
please test their package with the new webkit2gtk version in s-p-u
within the next 2 weeks.
5. If no regressions are reported, the update is released in the next
Debian 9 point release.

Here's a filtered list of rdeps: https://paste.debian.net/961084/

These updates are also being done across all Linux distributions so I
believe the risk of a regression with this process is fairly low.

If that process goes well, then we can consider how we will handle the
new major release later this year.

Later webkit2gtk updates

Based on the schedule of prior Debian point releases, I am concerned
about the longer gap between point releases later in Stretch's life. I
don't think it makes sense for Debian users to have to wait 4-5 months
for security updates. Maybe testing can still be done in s-p-u but
Security can push webkit2gtk versions from there out sooner.

Regressions from NOT updating
---
On the other hand, if we do not update webkit2gtk there will be
regressions too. A major recent example was Google changing their
websites which broke Google login (breaking a key part of GNOME Online
Accounts, Evolution, etc.) and the beta version of YouTube. This was
fixed a few weeks ago in 2.16.2 (and 2.14.7 but 2.14 is unsupported
now). https://launchpad.net/bugs/1687019

More diff info
---
The debdiffs themselves are difficult to read. Here are the 2.16.2 and
the 2.16.3 updates:
https://launchpadlibrarian.net/319531521/webkit2gtk_2.16.1-1_2.16.2-1.diff.gz
https://launchpadlibrarian.net/321119967/webkit2gtk_2.16.2-1_2.16.3-1.diff.gz

webkit2gtk still uses trac and svn and there's no git mirror yet:
https://trac.webkit.org/log/webkit/releases/WebKitGTK/webkit-2.16
To see colored diffs, just select the radio buttons next to the 2
commits you want to diff and click View Changes. I believe 2.16.2 was
216485 and 2.16.3 is 217366. To view individual commits, you have to
click the radio buttons for the current and the previous commits and
click View Changes.

Thanks,
Jeremy Bicha

Bug#863915: unblock: webkit2gtk/2.16.3-2

[Jeremy Bicha]

On Fri, Jun 2, 2017 at 3:58 AM, Adam D. Barratt
 wrote:
> On 2017-06-02 8:32, Adam D. Barratt wrote:
>> There's a huge leap from there to you assuming that SRM will be happy
>> to do this without any form of actual discussion. Unblock bugs are
>> *not* the way to have that discussion, and two weeks before release is
>> a really poor time to attempt to do so.

Respectfully, I am trying to do the best I know how to drive this
forward. We all want to keep Debian both stable and secure so let's
assume good faith here. If there is a better place to have a
discussion than this bug, let me know but it doesn't seem to me to be
a particularly wrong place for it either.

> Particularly given that the 2.15.0 release was 8 months ago now so there was
> plenty of time to have initiated a discussion, rather than leaving it until
> almost literally the last minute and assuming it would all be fine.

I apologize that I do not have a link to a public discussion, but my
understanding is that Alberto Garcia (berto), maintainer of webkit2gtk
in Debian and full-time paid developer for webkitgtk for Igalia, tried
repeatedly to get the Debian Security team to agree to allowing
stretch-security updates of webkit2gtk. If you're told "NO" enough
times, you eventually stop asking. Based on the work I've done to help
maintain webkit2gtk in Ubuntu, I'm trying to help Debian now.

On Fri, Jun 2, 2017 at 4:27 AM, Emilio Pozuelo Monfort  wrote:
> Could you list all the known regressions that resulted from these updates in
> Ubuntu? I think that would be an interesting data point for this discussion, 
> so
> that we can assess not just the number of regressions, but the severity of 
> them
> and how/if they were fixed (e.g. if upstream cared about these in the cases 
> that
> were reported to them, etc). If you can provide bug#, severity, and a timeline
> (e.g. webkit update to -proposed, webkit update to $distro, date of regression
> reported, regression fixed) that'd be helpful.

There have been no known significant regressions in Ubuntu stable
releases since Ubuntu started providing these webkit2gtk updates in
September when 2.10.9 was upgraded to 2.12.5.

Here are 2 significant regressions that did not affect Ubuntu stable releases:

2.12.4 did have a regression that "caused a hang in the network
process after a load failure". 2.12.4 was released August 24 and
2.12.5 fixing those problems was released September 5 (12 days later).

2.14.4 did have a regression in HiDPI support. 2.14.4 was released
February 10. The upstream bug was filed that same day. It was fixed in
upstream's svn repo February 13. 2.14.5 fixing that regression was
released February 15.
https://bugs.debian.org/855103
https://bugs.webkit.org/168128
https://mail.gnome.org/archives/distributor-list/2017-February/msg2.html
(public warning on February 13)

Thanks,
Jeremy

Bug#863915: unblock: webkit2gtk/2.16.3-2

[Carlos Alberto Lopez Perez]

On 02/06/17 14:47, Carlos Alberto Lopez Perez wrote:
>> Also it'd be nice to know what kind of automated testing is happening. I know
>> WebKit has an extensive test suite (including layout tests) that upstream
>> continuously runs for development series. I don't know if that's the same for
>> stable series though. Also can we enable the test suite in Debian?
>>
> On trunk (master) we have a extensive test coverage per each commit.
> We even have a bot testing that WebKitGTK+ always remains build-able
> both on Debian stable and Ubuntu LTS. Check our bots here:
> http://build.webkit.org/waterfall?category=GTK
> 
> All those bots are running Debian 9, except for the 32-bit and ARM bot
> that are running Debian 8 but they will be eventually upgraded to Debian
> 9 also.
> 
> (Also except the Debian and Ubuntu bots that are of course running
> Debian 8 and Ubuntu 16.04)
> 
> For the stable branch (that is what you are interested about I guess) we
> have a bot running all those tests here:
> 
> https://build-webkit.igalia.com/waterfall?category=GTK
> 
> This bot running the tests on the stable branch is currently running
> Debian 8 as base distro.
> We are happy to upgrade it to Debian 9 if you wish (is something we will
> eventually do in any case).
> 
> Note that those 9 failures are not really meaningful.
> Current layout test suite of WebKit has 47556 tests
> 
> If there is some more testing we can do to accommodate your needs, we
> will be happy to help.
> 
> Regards.
> 
> 

One note more:

The results of the layout tests are highly dependent on very specific
versions of the libraries we depend on. For example, different versions
of GTK+ or Cairo produce different results (this are usually small pixel
differences on the render of the webview).

So what we do, is to have the expected results of those tests specially
created for very specific versions of those libraries that we build with
JHBuild and we use it for testing.

If you try to run the layout test suite without using our internal
JHBuild you are going to get thousands of tests failures due to this
(they are not real failures, but the tooling is unable to detect that as
it can't know if a 1-pixel difference is because of a real failure or
because your version of GTK+ draws arrows/boxes/etc in a sightly
different way than the one was used to produce the expected result)

So, I don't think is flexible for Debian to run our layout test suite
unless you are willing to maintain your own list of expected results.

There are however other tests that you can run that don't depend on the
version of the libraries used like the jscore-tests or the API tests.
However, the tarballs we release are trimmed to remove all those testing
stuff from them. You would have to use the SVN repository for that,
which complicates things.


However, we are happy to keep our bot that runs all those tests for the
stable branch on Debian.




signature.asc
Description: OpenPGP digital signature

Bug#863915: unblock: webkit2gtk/2.16.3-2

[Carlos Alberto Lopez Perez]

On 02/06/17 10:27, Emilio Pozuelo Monfort wrote:
> Hi Jeremy,
> 
> On 01/06/17 23:15, Jeremy Bicha wrote:
>> Please unblock package webkit2gtk for inclusion in Debian 9.0.
>>
>> unblock webkit2gtk/2.16.3-2
>>
>> Justification
>> --
>> Three known publicized security vulnerabilities have been fixed in
>> 2.16.3: CVE-2017-2496, CVE-2017-2539 and CVE-2017-2510. For more
>> details about these and other recent security fixes, see [1].
>>
>> webkit2gtk follows GNOME's Release Schedule (new major updates in
>> March and September with bugfix updates in between). The 2.14 series
>> is no longer supported and will not be updated to fix those or future
>> security vulnerabilities.
>>
>> Background Info
>> 
>> Sadly, Debian's security packaging infrastructure is not set up to
>> test this kind of update very well. To provide a reasonable balance
>> between security for Debian 9 users and API stability for apps, the
>> current proposal [2] is to use Debian's s-p-u procedures and get these
>> updates into Debian point releases. This is a huge improvement over
>> Debian 8 where webkitgtk got only one early update and webkit2gtk was
>> only updated through backports. [3] [4] [5]
>>
>> To summarize a bit of the discussion on debian-devel, Ubuntu 16.04 LTS
>> has been receiving new webkit2gtk versions within about a week of
>> their release. Although regressions are possible, these have been
>> averted so far because Ubuntu tests the new major beta releases in the
>> development release of Ubuntu and because regressions are quickly
>> pointed out by users of more bleeding-edge distros (and these
>> regressions are quickly fixed!)
> 
> Could you list all the known regressions that resulted from these updates in
> Ubuntu? I think that would be an interesting data point for this discussion, 
> so
> that we can assess not just the number of regressions, but the severity of 
> them
> and how/if they were fixed (e.g. if upstream cared about these in the cases 
> that
> were reported to them, etc). If you can provide bug#, severity, and a timeline
> (e.g. webkit update to -proposed, webkit update to $distro, date of regression
> reported, regression fixed) that'd be helpful.
> 
> Also it'd be nice to know what kind of automated testing is happening. I know
> WebKit has an extensive test suite (including layout tests) that upstream
> continuously runs for development series. I don't know if that's the same for
> stable series though. Also can we enable the test suite in Debian?
> 

On trunk (master) we have a extensive test coverage per each commit.
We even have a bot testing that WebKitGTK+ always remains build-able
both on Debian stable and Ubuntu LTS. Check our bots here:
http://build.webkit.org/waterfall?category=GTK

All those bots are running Debian 9, except for the 32-bit and ARM bot
that are running Debian 8 but they will be eventually upgraded to Debian
9 also.

(Also except the Debian and Ubuntu bots that are of course running
Debian 8 and Ubuntu 16.04)

For the stable branch (that is what you are interested about I guess) we
have a bot running all those tests here:

https://build-webkit.igalia.com/waterfall?category=GTK

This bot running the tests on the stable branch is currently running
Debian 8 as base distro.
We are happy to upgrade it to Debian 9 if you wish (is something we will
eventually do in any case).

Note that those 9 failures are not really meaningful.
Current layout test suite of WebKit has 47556 tests

If there is some more testing we can do to accommodate your needs, we
will be happy to help.

Regards.




signature.asc
Description: OpenPGP digital signature

Bug#863915: unblock: webkit2gtk/2.16.3-2

[Emilio Pozuelo Monfort]

Hi Jeremy,

On 01/06/17 23:15, Jeremy Bicha wrote:
> Please unblock package webkit2gtk for inclusion in Debian 9.0.
> 
> unblock webkit2gtk/2.16.3-2
> 
> Justification
> --
> Three known publicized security vulnerabilities have been fixed in
> 2.16.3: CVE-2017-2496, CVE-2017-2539 and CVE-2017-2510. For more
> details about these and other recent security fixes, see [1].
> 
> webkit2gtk follows GNOME's Release Schedule (new major updates in
> March and September with bugfix updates in between). The 2.14 series
> is no longer supported and will not be updated to fix those or future
> security vulnerabilities.
> 
> Background Info
> 
> Sadly, Debian's security packaging infrastructure is not set up to
> test this kind of update very well. To provide a reasonable balance
> between security for Debian 9 users and API stability for apps, the
> current proposal [2] is to use Debian's s-p-u procedures and get these
> updates into Debian point releases. This is a huge improvement over
> Debian 8 where webkitgtk got only one early update and webkit2gtk was
> only updated through backports. [3] [4] [5]
> 
> To summarize a bit of the discussion on debian-devel, Ubuntu 16.04 LTS
> has been receiving new webkit2gtk versions within about a week of
> their release. Although regressions are possible, these have been
> averted so far because Ubuntu tests the new major beta releases in the
> development release of Ubuntu and because regressions are quickly
> pointed out by users of more bleeding-edge distros (and these
> regressions are quickly fixed!)

Could you list all the known regressions that resulted from these updates in
Ubuntu? I think that would be an interesting data point for this discussion, so
that we can assess not just the number of regressions, but the severity of them
and how/if they were fixed (e.g. if upstream cared about these in the cases that
were reported to them, etc). If you can provide bug#, severity, and a timeline
(e.g. webkit update to -proposed, webkit update to $distro, date of regression
reported, regression fixed) that'd be helpful.

Also it'd be nice to know what kind of automated testing is happening. I know
WebKit has an extensive test suite (including layout tests) that upstream
continuously runs for development series. I don't know if that's the same for
stable series though. Also can we enable the test suite in Debian?

Updating to supported webkit2gtk releases would be nice to keep it secure. OTOH
SRMs are (understandably) concerned about regressions. So let's see if we can
give them some more information so they can make an informed decision.

Thanks,
Emilio

Bug#863915: unblock: webkit2gtk/2.16.3-2

[Adam D. Barratt]

On 2017-06-02 8:32, Adam D. Barratt wrote:

There's a huge leap from there to you assuming that SRM will be happy
to do this without any form of actual discussion. Unblock bugs are
*not* the way to have that discussion, and two weeks before release is
a really poor time to attempt to do so.


Particularly given that the 2.15.0 release was 8 months ago now so there 
was plenty of time to have initiated a discussion, rather than leaving 
it until almost literally the last minute and assuming it would all be 
fine.


Regards,

Adam

Bug#863915: unblock: webkit2gtk/2.16.3-2

[Adam D. Barratt]

On 2017-06-01 22:15, Jeremy Bicha wrote:

Please unblock package webkit2gtk for inclusion in Debian 9.0.

[...]

Sadly, Debian's security packaging infrastructure is not set up to
test this kind of update very well.


I'm not entirely sure how you think p-u is better placed to do so, given 
the amount of visible testing packages from it get before a point 
release.



To provide a reasonable balance
between security for Debian 9 users and API stability for apps, the
current proposal [2] is to use Debian's s-p-u procedures and get these
updates into Debian point releases.


No. For the record, what Moritz actually said was:

"You're best technical bet would be to upgrade to new webkit releases in
stretch point releases, this would allow proper binNMUs and allow
people to testdrive via s-p-u. But that's up for the SRMs to
decide (and I doubt they want to deal with that kind of API
"stability" either)."

There's a huge leap from there to you assuming that SRM will be happy to 
do this without any form of actual discussion. Unblock bugs are *not* 
the way to have that discussion, and two weeks before release is a 
really poor time to attempt to do so.


Regards,

Adam