Re: Disabling whoopsie by default in the 12.04.1 release

[Steve Langasek]

Hi Seb,

On Mon, Aug 20, 2012 at 12:36:07PM +0200, Sebastien Bacher wrote:
> Le 20/08/2012 12:26, Evan Dandrea a écrit :
> >Apologies. It was not my intention to imply you accepted the
> >decision, just that your team picked a number you would be happy
> >with, and we came up with data that indicated we were below that.

> Right, we picked a "3 dialogs showing up in a week" as a max rate,
> the number you guys came up with is under that but:

> - you took the number on a 90 days period I think, knowing that most
> of the common issues would be stopped after 3 reports by apport.
> It's likely that the number of the first weeks of use is much higher
> than the one you came with and it's the one make the first
> impression (if people give up after a week because Ubuntu is too
> buggy we just lost them even if the annoyance would have gone down
> over time)

Sorry, but I feel like this is a case of moving the goal posts.  We
discussed what would be an acceptable per-week average.  I would not be ok
with drawing a line saying that the average user's *maximum* number of
crashes for any week must be 3 or below, because any user who in one week
had a single repeating crash, plus any single other crash, would miss this
limit.  If a crash is affecting users repeatedly, that's important
information to know, and we shouldn't discard this data to keep the user's
dialog count artificially low.

It seems to me that there are some answers the crashdb could give here that
might be useful:

 - What is the average number of crashes reported by a machine during the
   first week that the machine is seen?  (measuring the initial experience
   vs. the post-install experience)
 - What is the percentage of systems that have a high initial rate of
   crashes, which continue submitting crash reports after a week / a month /
   three months?  How does this percentage vary with the initial crash rate?
   (approximation of the effect of crash dialogs on user retention)
 - How has the initial crashiness changed over time / what is the effective
   crash rate if all 12.04.1 updates are applied?  (measuring the
   effectiveness of the 12.04.1 work on improving the user experience)

Perhaps having these answers would give us all greater confidence in whether
whoopsie is working the way we expect?  Are there other questions that you
think we should get answers to from the crashdb?

> - we still didn't explain that 90% drop showing in a lot of the
> reports' curves

That's not true at all; I gave you the explanation on IRC as part of the
discussion.  There is a bug in apport which causes it to randomly submit
incomplete crash reports, severely impacting our ability to correctly bucket
the information being submitted.  It does *not* impact the numbers of
reports submitted: any analysis of the perceived crashiness of the desktop
is unaffected by this bug.

The bug does have the following other implications:

 - The actual user experience benefit of fixing a crash shown on
   errors.ubuntu.com is roughly 10x what it would appear to be based on the
   graphs.  This means, for instance, that with approximately 75,000
   reports/day, the top crash of the day for 12.04, with a count of 342,
   probably accounts for 4% of the crash reports from users - not .4%.
 - Crashes in packages with heavyweight apport hooks are likely to be
   underrepresented in the list of top crashes, because they're more likely
   to have incomplete data submitted.  It's impossible to quantify this
   effect without just fixing the bug.
 - Infrequent crashes may fail to be bucketed at all, if the only crash
   reports sent for the crash are incomplete ones.  However, statistically
   speaking any such crashes are likely to be ones we aren't going to work
   on anyway because they have such a small impact.

Anyway, this is an important bug to fix and an SRU of apport is being
prepared for it.  This will improve our accuracy in collating data from
crash reports.  But it does *not* affect the accuracy of our raw numbers
regarding crash submissions.

> - the 0.4 report/week/user just doesn't match real life experience,
> nautilus and some other services generate a report every second
> logout, that alone should put us over that number if you consider
> users restart their computer once a day, and we didn't start
> counting any of the "real" issues...

That hasn't been my experience.  I've *never* seen a crash report from
nautilus on logout.  So it seems this is firmly in the realm of anecdotal
evidence, because we don't know whose experience is the typical one.

> Well anyway, as said let's move on, but I still think we took a
> non-decision to keep things the way they are based on approximative
> datas

It wasn't a non-decision at all.  It was an affirmative decision to not
change apport because the data does not support making a change.

All data is imperfect and should be questioned, and I've suggested above
some ways that this particular data can be refined if you think i

Re: Disabling whoopsie by default in the 12.04.1 release

[Martin Pitt]

Hello Evan,

Evan Dandrea [2012-08-20 10:41 +0100]:
> How can you programmatically tell that a daemon crashing has not affected
> the UI?

We can't. We just know that in a lot of cases it doesn't.

> I disagree that they do not help anyone.

I didn't say that. I just said that there are a lot of such crashes
which do not affect the UI and which don't get fixed in a stable
release.

> > In many cases the popup itself _is_ the mystery, though :-(
> >
> 
> Every other mainstream operating system has found a way to communicate to
> the user what just happened to their application.

I was specifically talking about crashes that do not affect
applications. As I said, I'm happy about the application crash popups,
I'd just reconsider the "Ubuntu has encountered an internal error"
popup, which does appear out of the blue.

> OS X does this:
> http://www.flickr.com/photos/skyzyx/6899505621/
> 
> Windows does this:
> http://elhombre.members.winisp.net/vista_watson01.png

These are both application related, and we do agree on that part.

> This is a proven design pattern. If the dialog text is confusing, lets
> focus on improving that.

They would still appear out of nowhere, are not SRU matter in many
cases, and not even visible in many cases. Changing dialog texts won't
help much with that, I'm afraid?

Thanks,

Martin
-- 
Martin Pitt| http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)

-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Martin Pitt]

Evan Dandrea [2012-08-20 10:19 +0100]:
> I think the less people we have working on a point release, the more
> valuable accurate data becomes.

Accurate, yes. But it becomes less useful, as the ratio of "user
annoyance" vs. "actual bug fixing" becomes a lot higher.

> > So perhaps one possible compromise would be to only show crashes for
> > packages in -proposed and -updates? This would limit the reports to
> > the set of packages that we are still actively working on and would
> > retain the whoopsie funcionality for pending SRUs, but would silence
> > it for the vast majority of packages which we don't fix any more in
> > precise anyway.
> 
> 
> I think this assumes that only packages we've just updated meet the
> threshold for further fixes. Each point release will bring large numbers of
> new users to the operating system, who's behavior may differ greatly from
> the existing people feeding crashes into http://errors.ubuntu.com.

Hence "compromise". :-)

> I do appreciate the effort to find a compromise though. We did ultimately
> decide to keep error reporting on for 12.04.1

OK, understood.

> I still don't understand why we're not fixing these applications.

It's not that we don't want to, but fixing these issues has never made
it to the top of the priority list. There are always bigger fish to
fry, and except for the apport/resulting confusion aspect, they are
really quite irrelevant.

> I worry this is just going to let us ignore the problem.

We don't say "we do not want to fix this", it's a matter of
prioritization of limited resources. :-)

Thanks,

Martin
-- 
Martin Pitt| http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)

-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Sebastien Bacher]

Le 20/08/2012 12:26, Evan Dandrea a écrit :
Apologies. It was not my intention to imply you accepted the decision, 
just that your team picked a number you would be happy with, and we 
came up with data that indicated we were below that.


Right, we picked a "3 dialogs showing up in a week" as a max rate, the 
number you guys came up with is under that but:


- you took the number on a 90 days period I think, knowing that most of 
the common issues would be stopped after 3 reports by apport. It's 
likely that the number of the first weeks of use is much higher than the 
one you came with and it's the one make the first impression (if people 
give up after a week because Ubuntu is too buggy we just lost them even 
if the annoyance would have gone down over time)


- we still didn't explain that 90% drop showing in a lot of the reports' 
curves


- the 0.4 report/week/user just doesn't match real life experience, 
nautilus and some other services generate a report every second logout, 
that alone should put us over that number if you consider users restart 
their computer once a day, and we didn't start counting any of the 
"real" issues...


Well anyway, as said let's move on, but I still think we took a 
non-decision to keep things the way they are based on approximative datas


Cheers,
Sebastien Bacher

--
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Evan Dandrea]

On Mon, Aug 20, 2012 at 11:01 AM, Sebastien Bacher wrote:

> Le 20/08/2012 11:19, Evan Dandrea a écrit :
>
>  We did ultimately decide to keep error reporting on for 12.04.1, as the
>> average number of errors per week met the requirement set forth by the
>> desktop team:
>>
> Hi,
>
> For the record I think that number is buggy (and I raised that concerns
> during laste week discussions), it seems artificially low and we identified
> several issues in the system without explaining all of them nor their
> impact (the 90% drop which happened for lot of bugs on 17/05/12 being one,
> the fact that bugs that fail retracing are not showing up on the lists, the
> fact that the first weeks stats seem much higher and that's the ones which
> give the first impression for users so that should be the number to
> consider for the choice, etc).
>
> I'm not happy about the decision and how it was taken (in retrospect I
> wish I had brought the topic in front of the TB rather to have a "what is
> best for Ubuntu" project's discussion), let's move on but I don't consider
> the "met the requirement set forth by the desktop team" true so let's
> refrain from making it look like the desktop team agreed on or supports the
> decision.
>

Apologies. It was not my intention to imply you accepted the decision, just
that your team picked a number you would be happy with, and we came up with
data that indicated we were below that.

Problems failing to retrace would in no way influence these numbers. They
still get counted.
-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Sebastien Bacher]

Le 20/08/2012 11:19, Evan Dandrea a écrit :
We did ultimately decide to keep error reporting on for 12.04.1, as 
the average number of errors per week met the requirement set forth by 
the desktop team:

Hi,

For the record I think that number is buggy (and I raised that concerns 
during laste week discussions), it seems artificially low and we 
identified several issues in the system without explaining all of them 
nor their impact (the 90% drop which happened for lot of bugs on 
17/05/12 being one, the fact that bugs that fail retracing are not 
showing up on the lists, the fact that the first weeks stats seem much 
higher and that's the ones which give the first impression for users so 
that should be the number to consider for the choice, etc).


I'm not happy about the decision and how it was taken (in retrospect I 
wish I had brought the topic in front of the TB rather to have a "what 
is best for Ubuntu" project's discussion), let's move on but I don't 
consider the "met the requirement set forth by the desktop team" true so 
let's refrain from making it look like the desktop team agreed on or 
supports the decision.


Cheers,
Sebastien Bacher

--
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Evan Dandrea]

On Mon, Aug 20, 2012 at 7:59 AM, Martin Pitt  wrote:

> Dylan McCall [2012-08-06 10:49 -0700]:
> > I don't run a computer lab, but I did upgrade someone's computer to
> > Ubuntu 12.04. A few days later, I felt like a total jerk as I stepped
> > him through disabling error popups using terminal commands.
>
> Admittedly I have done this to the Ubuntu box of my sister as well
> after she called me about all those popups (although I have ssh
> access, so it was a lot easier for me to disable it myself). First,
> she does not have any clue about computers and the unexpected popups
> confuse her, and second she only has a rather limited 3G connection
> and really shouldn't upload Jigabytes of crash dumps without her
> knowledge.
>

And we do not. Whoopsie only uploads reports when it can detect a non-3G
connection. It does this using a combination of GNetworkMonitor and
NetworkManager's DBus API.


> I felt bad when doing it, but after all she wants to use her computer,
> not spend her time with crash reports.
>
> For computer labs I expect that apport will be turned off wholesale
> anyway. There are enough HowTos in the interwebs to tell people how to
> do it, and with proxies/firewalls/or simply company policies about
> secrecy/privacy being around, we probably won't hear about many
> problems anyway.
>

There are very large tech companies that have expressed a desire to have
this on, but reporting to a server behind their firewall. They would then
forward issues up to us selectively. We do not have the resources for
implementing this in the near term, but I thought it worth mentioning.
-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Evan Dandrea]

Sorry Martin, I posted with the wrong address initially.

On Mon, Aug 20, 2012 at 7:41 AM, Martin Pitt  wrote:

> Matthew Paul Thomas [2012-08-07 10:47 +0100]:
> > But optimizing purely for the number of error prompts is the wrong
> > goal.
>
> I don't think that this is being done here. I would just like to
> suppress dialogs which are irrelevant for the user, and which are
> unlikely to ever get fixed in a stable release anyway (cf. crashes
> during logout, the rather large subset of daemon/python thread crashes
> which do not affect the UI, etc.). Because those won't ever go away,
> are a nuisance for the user, and don't help anyone.


How can you programmatically tell that a daemon crashing has not affected
the UI?

I disagree that they do not help anyone. I imagine the Ubuntu One team
wants to know about every time the syncdaemon fails. Equally, lets say we
have a program doing regular backups of the filesystem. The program
crashes. Okay, fine, it autorespawns. It crashes again. It never reaches
the point where it backs up all the user's data. We never know this because
we've decided that applications which recover through autorespawnning are
not worth our time.


> > We then have a choice between explaining what went wrong, or leaving
> > it a mystery.
>
> In many cases the popup itself _is_ the mystery, though :-(
>

Every other mainstream operating system has found a way to communicate to
the user what just happened to their application.

OS X does this:
http://www.flickr.com/photos/skyzyx/6899505621/

Windows does this:
http://elhombre.members.winisp.net/vista_watson01.png

There are a large subset of applications on iOS doing this:
http://www.hockeyapp.net/feature-overview
http://airbrake.io/pages/home
http://www.bugsense.com/

Android does this:
http://android-developers.blogspot.co.uk/2010/05/google-feedback-for-android.html

This is a proven design pattern. If the dialog text is confusing, lets
focus on improving that.


> I'm not arguing against showing the popup for applications which just
> crashed in the user's session. Those are fine, and as I said in
> LP#1033471 I think it's totally fine to turn off rate-limiting for
> this case.


Excellent.
-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Evan Dandrea]

On Mon, Aug 20, 2012 at 5:44 AM, Martin Pitt  wrote:

> I fully agree. However, this only catches one aspect of the problem.
> Of course this will be vastly useful for the SRUs that we'll do, but
> it will be mostly noise and false expectations for the crashes that we
> won't fix in stables (which will be the majority). After 12.04.1 there
> won't be a dedicated team any more, and all but the most common
> crashes will just stay as they are.
>

I think the less people we have working on a point release, the more
valuable accurate data becomes. With such small numbers attacking the
problem, I would hope the release team wants them picking from the top of a
severity-ordered list, rather than what they *think* is the most pressing.

So perhaps one possible compromise would be to only show crashes for
> packages in -proposed and -updates? This would limit the reports to
> the set of packages that we are still actively working on and would
> retain the whoopsie funcionality for pending SRUs, but would silence
> it for the vast majority of packages which we don't fix any more in
> precise anyway.


I think this assumes that only packages we've just updated meet the
threshold for further fixes. Each point release will bring large numbers of
new users to the operating system, who's behavior may differ greatly from
the existing people feeding crashes into http://errors.ubuntu.com. There
will likely be high-instance problems for packages that we did not issue an
update for, that we were not aware of prior to the release of 12.04.1, just
as this occurred for the 12.04 release.

I do appreciate the effort to find a compromise though. We did ultimately
decide to keep error reporting on for 12.04.1, as the average number of
errors per week met the requirement set forth by the desktop team:

http://irclogs.ubuntu.com/2012/08/16/%23ubuntu-release.html#t15:32

Of course, we should keep discussing how we can improve the situation with
actionable items. I'm working hard to get the multiple error reports in a
single dialog work done for 12.04.2 and 12.10:

https://wiki.ubuntu.com/ErrorTracker#When_there_are_multiple_simultaneous_errors

> > No, they are normal non technical users who get 5 prompts on login
> > > while they didn't start doing anything and when they box is working
> > > normally and in a clean state because we collect i.e random shutdown
> > > issues and are showing them those after next book.
> >
> > In practice, are the dialogs shown on login after shutdown the main
> problem?
> > If so, there might be a way to mitigate this particular problem without
> > hobbling whoopsie entirely.
>
> Apport 1.26 (since November 2011) made an attempt to ignore crashes
> which happen during shutdown (LP #460932), but by nature this is a
> rather heuristic and brittle detection. If it still happens too often,
> we should definitively make the filter more agressive. This is a
> particular class of crashes which are totally not SRU-worthy, and
> would normally not have any visual impact at all. Now they just lead
> to making session shutdown a lot longer (because it's hanging on the
> Apport and core dump I/O), and getting "old crashes" reports at the
> next login.
>

I still don't understand why we're not fixing these applications. If you
handle the exception of an object on the bus going away, you'll catch more
than just errors on shutdown.

I worry this is just going to let us ignore the problem. It will mean
discarding potentially important errors that happen to occur at shutdown or
only occur at shutdown.
-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Martin Pitt]

Dylan McCall [2012-08-06 10:49 -0700]:
> I don't run a computer lab, but I did upgrade someone's computer to
> Ubuntu 12.04. A few days later, I felt like a total jerk as I stepped
> him through disabling error popups using terminal commands.

Admittedly I have done this to the Ubuntu box of my sister as well
after she called me about all those popups (although I have ssh
access, so it was a lot easier for me to disable it myself). First,
she does not have any clue about computers and the unexpected popups
confuse her, and second she only has a rather limited 3G connection
and really shouldn't upload Jigabytes of crash dumps without her
knowledge.

I felt bad when doing it, but after all she wants to use her computer,
not spend her time with crash reports.

For computer labs I expect that apport will be turned off wholesale
anyway. There are enough HowTos in the interwebs to tell people how to
do it, and with proxies/firewalls/or simply company policies about
secrecy/privacy being around, we probably won't hear about many
problems anyway.

Martin

-- 
Martin Pitt| http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)

-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Martin Pitt]

Sebastien Bacher [2012-08-08 16:58 +0200]:
> Le 08/08/2012 16:31, Evan Dandrea a écrit :
> >>- password prompts locking your screen because an error happened to a
> >>service running with another user than yours
> >Is this happening automatically, or after you click on something?
> >
> It usually happens when a process from another user hits an issue,
> e.g start a guest session, closing it, you will like have one of
> those showing in your session

Apport's "get me the system crash reports" logic only looks at crashes
for UID < 500, i. e. for system users. I just realized that lightdm's
guest users are indeed system users, which would cause that. I think
we should fix that, I filed http://pad.lv/1038881 

>   trackers. (LP: #460932 )"
> 
> It seems that cut a number of those but doesn't cover all the cases,
> we need somebody to sit down and look at the cases where it's still
> happening and how we could cover them... (maybe just reject any
> issue with a timestamp before the session start?)

That would drop crashes which happen from cron, but these sound like a
corner case. For the purposes of a precise SRU, this would certainly
be a pinpointed, small, and safe change.

Martin

-- 
Martin Pitt| http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)

-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Martin Pitt]

Matthew Paul Thomas [2012-08-07 10:47 +0100]:
> But optimizing purely for the number of error prompts is the wrong
> goal.

I don't think that this is being done here. I would just like to
suppress dialogs which are irrelevant for the user, and which are
unlikely to ever get fixed in a stable release anyway (cf. crashes
during logout, the rather large subset of daemon/python thread crashes
which do not affect the UI, etc.). Because those won't ever go away,
are a nuisance for the user, and don't help anyone.

> We then have a choice between explaining what went wrong, or leaving
> it a mystery.

In many cases the popup itself _is_ the mystery, though :-(

I'm not arguing against showing the popup for applications which just
crashed in the user's session. Those are fine, and as I said in
LP#1033471 I think it's totally fine to turn off rate-limiting for
this case.

Thanks,

Martin
-- 
Martin Pitt| http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)

-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Martin Pitt]

Hello all,

sorry for the late answer, just returned from holidays.

Steve Langasek [2012-08-06 23:14 -0700]:
> In an earlier message, Martin spoke of whoopsie being a "great experiment".

For the "experiment" part I was referring to keeping Apport enabled in
a stable release. We did not really know how many and which kind of
issues we would get before we actually did it. Now we have a much
better idea, and besides those numbers we now also know how much
effort it is to keep up with fixing the "top ten" crashers, as well as
the situations where the crash popups are useless and unnerving (e. g.
crashes that happen during logout, multiple system crashes from
respawning services, etc.)

> If we can't quickly answer these questions, I think that's a strong argument
> for turning whoopsie off *until we can*, with the intent of re-enabling it
> once we're sure that we're on track.  Yes, that would be a setback for
> gathering data about the most severe bugs; but the doubt here is about what
> it's costing us (in terms of user experience) to gather that data and
> whether we're making effective use of it in practice, and that's something
> it's really important for us to know.  Just as long as we bear in mind that
> the goal is to work out the kinks and get it re-enabled - among other
> things, whoopsie is crucial to us improving our SRU process, and that
> definitely requires it to be on in stable releases.

I fully agree. However, this only catches one aspect of the problem.
Of course this will be vastly useful for the SRUs that we'll do, but
it will be mostly noise and false expectations for the crashes that we
won't fix in stables (which will be the majority). After 12.04.1 there
won't be a dedicated team any more, and all but the most common
crashes will just stay as they are.

So perhaps one possible compromise would be to only show crashes for
packages in -proposed and -updates? This would limit the reports to
the set of packages that we are still actively working on and would
retain the whoopsie funcionality for pending SRUs, but would silence
it for the vast majority of packages which we don't fix any more in
precise anyway.

> > No, they are normal non technical users who get 5 prompts on login
> > while they didn't start doing anything and when they box is working
> > normally and in a clean state because we collect i.e random shutdown
> > issues and are showing them those after next book.
> 
> In practice, are the dialogs shown on login after shutdown the main problem? 
> If so, there might be a way to mitigate this particular problem without
> hobbling whoopsie entirely.

Apport 1.26 (since November 2011) made an attempt to ignore crashes
which happen during shutdown (LP #460932), but by nature this is a
rather heuristic and brittle detection. If it still happens too often,
we should definitively make the filter more agressive. This is a
particular class of crashes which are totally not SRU-worthy, and
would normally not have any visual impact at all. Now they just lead
to making session shutdown a lot longer (because it's hanging on the
Apport and core dump I/O), and getting "old crashes" reports at the
next login.

Thanks,

Martin

-- 
Martin Pitt| http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)


signature.asc
Description: Digital signature
-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Evan Dandrea]

On Thu, Aug 2, 2012 at 10:31 PM, Sebastien Bacher  wrote:
> - some of our engineers have issues login into the system to get access to
> the infos they need to work on the bugs, that situation is still not
> resolved after some months

This is fixed in trunk and is RT 55322: https://portal.admin.canonical.com/55322

Once landed, you will not be prompted again after the first login. It
should also resolve the issues that Didier was seeing.

Given that the webops team is managing a datacenter move today and
this weekend, I don't expect this to land until next week.

-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Sebastien Bacher]

Le 13/08/2012 17:20, Evan Dandrea a écrit :
- If the linked bug is marked as completed, but the Last seen column 
contains the most recent published version of the package, the Last 
seen column will be marked red. This is to indicate a *possible* 
regression. I'm still playing with this. Because Invalid bugs are also 
interpreted as complete, and because we only consider a bug complete 
when all of its bug tasks are complete, this is currently wrong in a 
few places. I was more interested in seeing how it presented, however.


There seems to be a bug in that code, takes sessioninstaller: 
https://launchpad.net/bugs/848605

- the issue got fixed in 0.20+bzr128-0ubuntu1.1
- precise-updates has that version, same for quantal
- the "last seen" column has "0.20+bzr128-0ubuntu1", the detailled 
report page confirms that


but it's marked in red?

Cheers,
Sebastien Bacher

--
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Sebastien Bacher]

Le 13/08/2012 17:20, Evan Dandrea a écrit :

In the most common problems table:
- If a linked bug report is marked as completed, the entire row will 
be greyed out (but still clickable).
- If there is no linked bug report or the linked bug report is not 
marked as completed, but the version in Last seen column is *not* the 
most recent, just the Last seen column will be greyed out. This 
implies that because we have not seen a newer version with the 
problem, it is no longer present.
- If the linked bug is marked as completed, but the Last seen column 
contains the most recent published version of the package, the Last 
seen column will be marked red. This is to indicate a *possible* 
regression. I'm still playing with this. Because Invalid bugs are also 
interpreted as complete, and because we only consider a bug complete 
when all of its bug tasks are complete, this is currently wrong in a 
few places. I was more interested in seeing how it presented, however.


Hey Evan,

Those are excellent news, the improvements make a real difference to the 
side usability, we finally can easily sort out what issues on the main 
page got addressed and those that still need work!


Thanks,
Sebastien Bacher

--
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Evan Dandrea]

On Fri, Aug 10, 2012 at 4:38 PM, Evan Dandrea  wrote:

> On Thu, Aug 9, 2012 at 9:46 AM, Sebastien Bacher wrote:
>
>>  * did you say that fixed bugs should be colored differently in that
>> version?
>>
>
> Not fixed bugs, but problem pages that you've already visited. I'll try to
> find time this weekend to land the branch that makes problems that are
> likely to be fixed (they have yet to show up in the most recent version of
> the package) greyed out but still clickable.
>

As promised, I sorted this over the weekend and thanks to some quick work
by jjo, it's now live:

http://errors.ubuntu.com

In the most common problems table:
- If a linked bug report is marked as completed, the entire row will be
greyed out (but still clickable).
- If there is no linked bug report or the linked bug report is not marked
as completed, but the version in Last seen column is *not* the most recent,
just the Last seen column will be greyed out. This implies that because we
have not seen a newer version with the problem, it is no longer present.
- If the linked bug is marked as completed, but the Last seen column
contains the most recent published version of the package, the Last seen
column will be marked red. This is to indicate a *possible* regression. I'm
still playing with this. Because Invalid bugs are also interpreted as
complete, and because we only consider a bug complete when all of its bug
tasks are complete, this is currently wrong in a few places. I was more
interested in seeing how it presented, however.

I have another deployment in the pipeline that speeds this request up by
paying attention to the etags used by Launchpad and feeding it gzipped data.
-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Evan Dandrea]

On Fri, Aug 10, 2012 at 4:38 PM, Evan Dandrea  wrote:

> * clicking on some bugs give an error page,
>>
>> "  Exception Type: IndexError  Exception Value:
>>
>> list index out of range
>>
>>   Exception Location: /srv/
>> errors.ubuntu.com/production/errors/cassandra.py in
>> get_traceback_for_bucket, line 147  ..."
>>
>> example, try the gnome-control-center one with 26 reports today
>>
>
> Yes, that's only happening to problems that have failed to retrace. I'm
> looking into it.
>

This is fixed in trunk and should be picked up with the rest of RT 55227.
-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Evan Dandrea]

On Wed, Aug 8, 2012 at 3:24 PM, Evan Dandrea  wrote:
> Well, we have two options there as far as I can tell. We can either
> disable crash reporting during logout or shutdown, or we can better
> handle the presentation of those errors on subsequent login.
>
> The latter is tackled by the multiple simultaneous errors dialog that
> Matthew designed. As mentioned in this thread, I've started
> implementing this with the hope of getting it landed by the end of the
> week:
>
> https://wiki.ubuntu.com/ErrorTracker#When_there_are_multiple_simultaneous_errors

Now that I've been working on this for a few days, it seems
increasingly unlikely that we can reasonably land it by the week's
end. Because apport is architected strongly around an idea of
processing a single report, this is requiring major changes to the GTK
frontend with added complexity around keeping two single instances for
the multiple system errors and multiple application errors dialogs.

I'm going to press on, but I'd really like Martin to review this and
let it settle in quantal first as it's a big change after all.

-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Evan Dandrea]

On Thu, Aug 9, 2012 at 9:46 AM, Sebastien Bacher  wrote:

>  Le 09/08/2012 10:22, Evan Dandrea a écrit :
>
>  This has now landed:
> http://errors.ubuntu.com
>
>  Thanks, that's a really nice improvement!
>

Cheers! Apologies again. This should've landed weeks ago, but was
complicated by my holidays and the python-tastypie dependency chain.


>  - The individual problem pages now have their own graph, showing the
> number of instances of the problem over time.
>
>  There is something weird there, did we change something in the system?
> I've look at some 15 bugs and they all go up from release to 05/17/12 then
> drop to 10% of what they were at this point and stay flat ... it's really
> surprised that all bugs seem to have that drop, even the ones that on
> packages that got no update.
>

Yes, well spotted. They all drop off on May 18th. I've started digging into
this today but do not have any solid answers yet. We don't seem to be
receiving any fewer reports.

 - These pages also have a table showing the version by version
> breakdown of instances of this problem.
>
>  Excellent, the instance by version made my day, it's an amazing
> improvement to confirm that bugs got fixed in the current version
>

YAY! Very glad it's helping.

>  - Brian Murray made it so that visited problem pages use a different
> link color. A small but sorely needed change.
> - You can now specify a date range for the most common problems table.
> This is a bit fiddly at the moment. I'm working on fixing it.
> - Problems that failed to retrace appear in the most common problems
> table. There appears to be an issue with the retracers at the moment.
> I'm looking into it.
>
>  Some other issues:
>
> * I can't change the "Most common problems in" combo to "the past month",
> it just gives me a "An error occurred while trying to load the most common
> problems." ... it's kind of an important issue since the "today" view is
> too narrowed to get a good picture usually
>

I've fixed this in trunk. I'm going to request a deployment for that today.


> * clicking on some bugs give an error page,
>
> "  Exception Type: IndexError  Exception Value:
>
> list index out of range
>
>   Exception Location: /srv/
> errors.ubuntu.com/production/errors/cassandra.py in
> get_traceback_for_bucket, line 147  ..."
>
> example, try the gnome-control-center one with 26 reports today
>

Yes, that's only happening to problems that have failed to retrace. I'm
looking into it.


> * did you say that fixed bugs should be colored differently in that
> version?
>

Not fixed bugs, but problem pages that you've already visited. I'll try to
find time this weekend to land the branch that makes problems that are
likely to be fixed (they have yet to show up in the most recent version of
the package) greyed out but still clickable.

Thanks for the work, overall that update is a great step for those using
> the site ;-)
>

That's grand news. Thanks for the kind words.
-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Evan Dandrea]

On Tue, Aug 7, 2012 at 10:15 PM, Steve Langasek
 wrote:
> Ok.  How soon could we expect the fix for that bug to be rolled out?  Do I
> understand correctly that this can't be done quickly as a one-time query
> because the database structure needs to be updated first?

As mentioned out of band, this cannot be done quickly, but can be done
using the existing data we have. As with most everything else, no
client side changes are needed.

I had to first create a mapping back from individual crash reports to
the system identifiers. This took about 12 hours as we do not yet have
Hadoop up and running.

I've since written code to back-populate the per-release lists of
unique system identifiers so we can grab 90 days worth and then reduce
it down to the unique set after each day; however, in its current form
it's proving a bit much for the Cassandra nodes to handle. I've tuned
it to be a bit less aggressive (and thus slower), but the webops team
are currently fighting other fires and I'd prefer to have their full
attention should it start setting off their pagers again.

Matthew and Robert meanwhile have been discussing a more accurate
means of calculating the "if all updates were installed" lines. As
this still requires some mathematical verification, we're tabling that
particular feature of the website until we're more confident in our
approach.

-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Sebastien Bacher]

Le 09/08/2012 10:22, Evan Dandrea a écrit :


This has now landed:

http://errors.ubuntu.com

Thanks, that's a really nice improvement!


- The individual problem pages now have their own graph, showing the
number of instances of the problem over time.
There is something weird there, did we change something in the system? 
I've look at some 15 bugs and they all go up from release to 05/17/12 
then drop to 10% of what they were at this point and stay flat ... it's 
really surprised that all bugs seem to have that drop, even the ones 
that on packages that got no update.



- These pages also have a table showing the version by version
breakdown of instances of this problem.
Excellent, the instance by version made my day, it's an amazing 
improvement to confirm that bugs got fixed in the current version

- Brian Murray made it so that visited problem pages use a different
link color. A small but sorely needed change.
- You can now specify a date range for the most common problems table.
This is a bit fiddly at the moment. I'm working on fixing it.
- Problems that failed to retrace appear in the most common problems
table. There appears to be an issue with the retracers at the moment.
I'm looking into it.

Some other issues:

* I can't change the "Most common problems in" combo to "the past 
month", it just gives me a "An error occurred while trying to load the 
most common problems." ... it's kind of an important issue since the 
"today" view is too narrowed to get a good picture usually


* clicking on some bugs give an error page,

"
Exception Type: IndexError
Exception Value:

list index out of range

Exception Location: 
/srv/errors.ubuntu.com/production/errors/cassandra.py in 
get_traceback_for_bucket, line 147


..."

example, try the gnome-control-center one with 26 reports today

* did you say that fixed bugs should be colored differently in that version?


Thanks for the work, overall that update is a great step for those using 
the site ;-)


Sebastien Bacher



-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Evan Dandrea]

On Mon, Aug 6, 2012 at 11:04 AM, Evan Dandrea  wrote:
>> - it's not possible to filter out issues which have been resolved from the
>> list (so it's hard to know what has been worked on and what needs work
>> still)

*snip*

> In the iteration of the website that's in the process of being
> deployed (RT 54702), on each problem page you can see a table with the
> breakdown of version numbers and the number of instances of that
> problem. If the version you just uploaded does not ever appear, then
> you know the problem is fixed.

*snip*

>> - it's not possible to say what issue happens to what version and get stats
>> of instances of the bugs by version, i.e to get datas on whether the
>> situation improved or not for a bug
>
> As mentioned above, this is landing with RT 54702.

This has now landed:

http://errors.ubuntu.com

- On the average errors per day graph on the front page there are now
lines for each Ubuntu release, rather than one line for both 12.04 and
12.10. This still uses the old calculation of total crashes seen total
divided by total unique users seen today. However, my update of the
database to include the information we need to implement the 90 day
moving total of unique users
(https://bugs.launchpad.net/daisy/+bug/1033913) finished overnight. So
I'll hopefully land a better graph today or tomorrow.
- The individual problem pages now have their own graph, showing the
number of instances of the problem over time.
- These pages also have a table showing the version by version
breakdown of instances of this problem.
- Brian Murray made it so that visited problem pages use a different
link color. A small but sorely needed change.
- You can now specify a date range for the most common problems table.
This is a bit fiddly at the moment. I'm working on fixing it.
- Problems that failed to retrace appear in the most common problems
table. There appears to be an issue with the retracers at the moment.
I'm looking into it.

There were a few stumbles around getting the dependencies for
python-tastypie built for lucid. A massive thanks to jjo, who did much
of the heavy lifting there.

-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Sebastien Bacher]

Le 08/08/2012 16:31, Evan Dandrea a écrit :

- password prompts locking your screen because an error happened to a
service running with another user than yours

Is this happening automatically, or after you click on something?

It usually happens when a process from another user hits an issue, e.g 
start a guest session, closing it, you will like have one of those 
showing in your session



- we don't know if users make the connection between the issue and those
dialogs (what if gedit hits an issue when closing it, then apport takes a
minute to process the datas and then you get a dialog ... by the time you
moved to something else and wonder wth you get a gedit error when you
stopped using it for a minute)

If this is proving to be a problem, I'm sure we can find optimizations
to apport in R. Let me know if it is.
I'm not sure how to figure it out, but for sure I see often delay here 
between an issue and the dialog telling about it



That would be a good step indeed, we discussed that in the past but that's
not easy to do unfortunately and we didn't see so much progress on that
front so far but if somebody has good ideas how to do it in an reliable way,
or time to work on that, please let us know

Can you elaborate on the discussion that's taken place around this idea?

Could we not modify the session manager to write the graphical login
and logout (started) time to disk? Then, if a crash occurs between the
logout time and login time, we wait to report it until we have a crash
that's after the login time.

Some work was done on the past in
https://bugs.launchpad.net/ubuntu/+source/apport/+bug/460932

from the bug

"so we reject a crash if:

- $DBUS_SESSION_BUS_ADDRESS exists, but we get a d-bus error when trying 
to connect to it (shutting down session bus)

- org.gnome.SessionManager exists on the bus, but the call fails
- calling IsSessionRunning() returns false (being shut down)"

"- Ignore a crash if gnome-session is running and says that the 
session is

  being shut down. These often die because X.org or other services are
  going away, are usually harmless, and just cause a lot of clutter 
in bug

  trackers. (LP: #460932 )"

It seems that cut a number of those but doesn't cover all the cases, we 
need somebody to sit down and look at the cases where it's still 
happening and how we could cover them... (maybe just reject any issue 
with a timestamp before the session start?)


Cheers,
Sebastien Bacher
-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Evan Dandrea]

Sent using the wrong address. Apologies for the noise.

On Tue, Aug 7, 2012 at 12:23 PM, Sebastien Bacher  wrote:
> - dialogs that appears out of nowhere while working and where you didn't get
> an issue (let's make an example "oneconf regular index update failed in
> background"), the dialog doesn't guide the user or explain anything, it just
> creates confusion "what is oneconf", "is the fact that it failed an issue",
> "what should I do", "what does it mean for my system"

It is my understanding that this is precisely why we have a separate
dialog for system errors:

https://wiki.ubuntu.com/ErrorTracker#os-crash

If you're seeing the regular application error dialog for oneconf
despite it not having a desktop file, that's a bug. Please report it.

> - dialogs that appears after login but don't concern the current session,
> those don't explain anything or are not logically connected to the issue
> which happened in the previous session or at shutdown

Well, we have two options there as far as I can tell. We can either
disable crash reporting during logout or shutdown, or we can better
handle the presentation of those errors on subsequent login.

The latter is tackled by the multiple simultaneous errors dialog that
Matthew designed. As mentioned in this thread, I've started
implementing this with the hope of getting it landed by the end of the
week:

https://wiki.ubuntu.com/ErrorTracker#When_there_are_multiple_simultaneous_errors

> - password prompts locking your screen because an error happened to a
> service running with another user than yours

Is this happening automatically, or after you click on something?

Fixing Apport to have a privileged DBus backend or to better handle
the password prompts through some other means is definitely on the
radar:

https://bugs.launchpad.net/ubuntu/+source/apport/+bug/871662

> - we don't know if users make the connection between the issue and those
> dialogs (what if gedit hits an issue when closing it, then apport takes a
> minute to process the datas and then you get a dialog ... by the time you
> moved to something else and wonder wth you get a gedit error when you
> stopped using it for a minute)

If this is proving to be a problem, I'm sure we can find optimizations
to apport in R. Let me know if it is.

>> If we can isolate particular classes of error that we can be confident
>> don't need explaining, then great! Let's not show an error prompt for
>> those. (We could batch up the error reports, and send them next time
>> an error occurs that does need to be explained.) For example, perhaps
>> you could describe how to identify errors that happened at logout or
>> shutdown.
>>
> That would be a good step indeed, we discussed that in the past but that's
> not easy to do unfortunately and we didn't see so much progress on that
> front so far but if somebody has good ideas how to do it in an reliable way,
> or time to work on that, please let us know

Can you elaborate on the discussion that's taken place around this idea?

Could we not modify the session manager to write the graphical login
and logout (started) time to disk? Then, if a crash occurs between the
logout time and login time, we wait to report it until we have a crash
that's after the login time.

-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Jeremy Bicha]

On 8 August 2012 09:04, Sebastien Bacher  wrote:
> Hey again,
>
> I just crossed that online I figured I would point it as one of the examples
> I was making reference to:
> http://www.techdrivein.com/2012/08/how-to-disable-system-program-problem.html

And here's another one from a pretty popular Ubuntu blog:

http://www.webupd8.org/2012/06/how-to-get-rid-of-internal-system-error.html

Jeremy

-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Sebastien Bacher]

Le 07/08/2012 08:14, Steve Langasek a écrit :

because the information available remains rather incomplete.  I have not
heard the same feedback that you have about precise being buggy, and I'm not
sure that's actually representative of users' experience

Hey again,

I just crossed that online I figured I would point it as one of the 
examples I was making reference to:

http://www.techdrivein.com/2012/08/how-to-disable-system-program-problem.html

Cheers,
Sebastien Bacher

--
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Steve Langasek]

On Tue, Aug 07, 2012 at 05:12:44PM +0100, Evan Dandrea wrote:
> On Tue, Aug 7, 2012 at 7:14 AM, Steve Langasek
>  wrote:
> > Evan, can you provide some clarity here?  How many users is this average
> > taken from?

> It's currently using the total number of unique users for the day period:

> [default@crashdb] get Counters [utf8('users:Ubuntu 12.04')];
> ...
> => (counter=20120801, value=54191)
> => (counter=20120802, value=52659)
> => (counter=20120803, value=51273)
> => (counter=20120804, value=45861)
> => (counter=20120805, value=48042)
> => (counter=20120806, value=54958)
> => (counter=20120807, value=33539)

> This is wrong, as you've pointed out. Matthew has filed a bug for it here:
> https://bugs.launchpad.net/daisy/+bug/1033913

> Fixing this will not require any client-side changes.


> In case you were asking that in terms of the average number of crashes
> per day per user:
> Mean: 1.44842
> Standard deviation: 0.01867

Yep, that's the one I wanted, thanks :)

> > What's the turnover of users reporting crashes from day to day?  (I.e., it's
> > quite different if the same 5000 users are reporting 1.4 crashes per day,
> > than if 5000 *unique* users are reporting 1.4 crashes each day.) These are
> > the kinds of information I think we need to have in order to make an
> > informed decision.
> 
> We currently only count unique users per-release over the day period.
> Matthew's bug will have us tracking the unique users seen in the past
> 90 day period. If there's some other interval you'd like to track, do
> let me know. As before, this will require no client-side changes.

Ok.  How soon could we expect the fix for that bug to be rolled out?  Do I
understand correctly that this can't be done quickly as a one-time query
because the database structure needs to be updated first?

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: Digital signature
-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Reducing dialogues rate (was: Re: Disabling whoopsie by default in the 12.04.1 release)

[Evan Dandrea]

On Tue, Aug 7, 2012 at 5:28 PM, Steve Langasek
 wrote:
> On Tue, Aug 07, 2012 at 10:56:12AM +0200, Sebastien Bacher wrote:
>> If you log in and 5 reports are
>> "waiting" on the disk we should probably not have 5 dialogs
>> displaying in sequence...
>
> I agree.  And I understand this is now being worked on.

As discussed in the other email thread, I bumped this to the top of my
list. I've been working on it for much of the day, somewhat
complicated by GtkApplication not doing what I needed (releasing the
singleton while still running). Making steady progress though, and I
hope to have it ready before Friday.

https://wiki.ubuntu.com/ErrorTracker#multiple

-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Reducing dialogues rate (was: Re: Disabling whoopsie by default in the 12.04.1 release)

[Steve Langasek]

On Tue, Aug 07, 2012 at 10:56:12AM +0200, Sebastien Bacher wrote:
> Le 07/08/2012 08:40, Steve Langasek a écrit :
> >that data that aren't giving users a bad impression (if indeed that's what's
> >happening).  For instance, what if we were to only pop up the whoopsie
> >prompt for every fifth crash on a user's system on a stable release?  The
> >per-user dialogue rate would be slashed by 80% (er, except for the fact that
> That would be good to reduce the spamming but it would also breaks
> what Evan and Matthew consider one of the main features, telling
> users about problem when they happen and gives an easy way to
> restart the applications when it closed. We could maybe do something
> like that on session start though? If you log in and 5 reports are
> "waiting" on the disk we should probably not have 5 dialogs
> displaying in sequence...

I agree.  And I understand this is now being worked on.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: Digital signature
-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Evan Dandrea]

On Tue, Aug 7, 2012 at 7:14 AM, Steve Langasek
 wrote:
> Evan, can you provide some clarity here?  How many users is this average
> taken from?

It's currently using the total number of unique users for the day period:

[default@crashdb] get Counters [utf8('users:Ubuntu 12.04')];
...
=> (counter=20120801, value=54191)
=> (counter=20120802, value=52659)
=> (counter=20120803, value=51273)
=> (counter=20120804, value=45861)
=> (counter=20120805, value=48042)
=> (counter=20120806, value=54958)
=> (counter=20120807, value=33539)

This is wrong, as you've pointed out. Matthew has filed a bug for it here:
https://bugs.launchpad.net/daisy/+bug/1033913

Fixing this will not require any client-side changes.

> What's the standard deviation on the number of crashes per day?

Given the data we have specifically for 12.04, excluding today:

[default@crashdb] get Counters [utf8('oopses:Ubuntu 12.04')];
=> (counter=20120720, value=77821)
=> (counter=20120721, value=70931)
=> (counter=20120722, value=67626)
=> (counter=20120723, value=81136)
=> (counter=20120724, value=82249)
=> (counter=20120725, value=79902)
=> (counter=20120726, value=81455)
=> (counter=20120727, value=75325)
=> (counter=20120728, value=69669)
=> (counter=20120729, value=72365)
=> (counter=20120730, value=80259)
=> (counter=20120731, value=77617)
=> (counter=20120801, value=77398)
=> (counter=20120802, value=75483)
=> (counter=20120803, value=73580)
=> (counter=20120804, value=67283)
=> (counter=20120805, value=70334)
=> (counter=20120806, value=78391)

Mean: 75490.2
Standard deviation: 4858.3275

In case you were asking that in terms of the average number of crashes
per day per user:
Mean: 1.44842
Standard deviation: 0.01867

> What's the turnover of users reporting crashes from day to day?  (I.e., it's
> quite different if the same 5000 users are reporting 1.4 crashes per day,
> than if 5000 *unique* users are reporting 1.4 crashes each day.) These are
> the kinds of information I think we need to have in order to make an
> informed decision.

We currently only count unique users per-release over the day period.
Matthew's bug will have us tracking the unique users seen in the past
90 day period. If there's some other interval you'd like to track, do
let me know. As before, this will require no client-side changes.

-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Matthew Paul Thomas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Dylan McCall wrote on 06/08/12 18:49:
> 
> On Mon, Aug 6, 2012 at 6:20 AM, Matthew Paul Thomas
>  ...
>> 
>> That isn't true, unless today is a freak exception. Right now,
>> out of the 50 most common errors, only 17 are from services. The
>> rest are from applications.
> 
> Isn't that _reported_ errors? Do you have any numbers for error
> popups that have been dismissed?

Dismissing the error alert sends an error report by default. Because
no extra effort is required, we can be confident that errors reported
are fairly representative of errors occurring.

Errors go unreported if you log out without dismissing the alert, if
you uncheck the checkbox before dismissing the alert, if your admin
has blocked error reporting in System Settings, or if there's a bug in
Apport or Whoopsie. We don't have numbers on unreported errors because
they are, you know, unreported. :-)

> Personally, I almost always dismiss the system error popups. They
> are vaguely worded and usually for the same problem in
> mission-control. The application errors, on the other hand, are
> upfront about what is broken. It's likely I have just seen that 
> application crashing. I know (and care about) whatever is going
> on. Oh, it also helps that lots of background stuff loves to crash
> during shutdown / suspend / resume (resulting in crash popups when
> I log in), while application crash popups are at slightly less
> annoying, and more meaningful, moments. I'm willing to accept that
> I could be an exception, but I suspect the numbers of reported
> errors might be biased in this way.

Until yesterday I didn't know that error alerts were appearing for
errors that occurred in the previous session. It isn't appropriate to
prompt about those (unless they directly caused the session exit). So
what we need now is a way to distinguish those from the rest.


> ...
> 
> PS: For what it's worth, Microsoft's Action Centre thing from
> Windows 7 might actually be an interesting model for this. It is
> mostly a useless thing that nags people to set up backups, but it
> also has a nice bit that lists collected system errors. It's useful
> to actually diagnose problems that are occurring. In Ubuntu all we
> seem to do is nag the user to report errors (which are promptly
> forgotten), and there's no real sense of getting something back for
> the trouble. What if that data was aggregated locally, too, so a
> user could see that a particular component is crashing really
> frequently? Or report issues from crash popups that had been
> previously dismissed? It might make the whole thing more agreeable
> ;)

We've planned a "Show Previous Reports" button in the settings panel,
that goes to a Web page listing the reports you've sent.


And eventually, the error alert will be able to invite you to install
updates because an update is available that fixes the specific problem.


- -- 
mpt

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAlAg/sIACgkQ6PUxNfU6ecq/+ACfXHxAUIZXTyzKyIa+Cb3briOA
erIAoMLny/34XMrEc5KcoCvceRRSrCvO
=hkBQ
-END PGP SIGNATURE-

-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Sebastien Bacher]

Le 07/08/2012 11:47, Matthew Paul Thomas a écrit :



But optimizing purely for the number of error prompts is the wrong
goal. The situations we're discussing are situations where *something
has already gone wrong*. We then have a choice between explaining what
went wrong, or leaving it a mystery.

I disagree with that, it's half of the story.

The other side of the coin is:

- dialogs that appears out of nowhere while working and where you didn't 
get an issue (let's make an example "oneconf regular index update failed 
in background"), the dialog doesn't guide the user or explain anything, 
it just creates confusion "what is oneconf", "is the fact that it failed 
an issue", "what should I do", "what does it mean for my system"


- dialogs that appears after login but don't concern the current 
session, those don't explain anything or are not logically connected to 
the issue which happened in the previous session or at shutdown


- password prompts locking your screen because an error happened to a 
service running with another user than yours


I would argue that those are not helping the user, quite the contrary


Now, you propose that so few of the crashes in Ubuntu 12.04.1 need to
be explained, that *none* of them should be. But you haven't provided
any data to support this. The claim that most crashes are in services
isn't supported by the evidence. (Of the most common reported errors,
yesterday 17/50 were; today only 7/50 were.)


First qualifying my position as "none should be explained" is 
misleading, I would really much to have real user facing issues 
explained in a way which makes sense to users, I challenge that our 
current system does that though


I don't have hard datas, but you don't have either...does anyone has 
ideas how to get better numbers?


What I can say from those numbers is that we:
- don't know how many got displayed is a reasonable timeframe to the 
user to have a meaningful connexion to the issue (opposed to e.g next login)

- don't know how many have an user visible impact
- we don't know if users make the connection between the issue and those 
dialogs (what if gedit hits an issue when closing it, then apport takes 
a minute to process the datas and then you get a dialog ... by the time 
you moved to something else and wonder wth you get a gedit error when 
you stopped using it for a minute)




If we can isolate particular classes of error that we can be confident
don't need explaining, then great! Let's not show an error prompt for
those. (We could batch up the error reports, and send them next time
an error occurs that does need to be explained.) For example, perhaps
you could describe how to identify errors that happened at logout or
shutdown.

That would be a good step indeed, we discussed that in the past but 
that's not easy to do unfortunately and we didn't see so much progress 
on that front so far but if somebody has good ideas how to do it in an 
reliable way, or time to work on that, please let us know


Cheers,
Sebastien Bacher

--
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Matthew Paul Thomas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Matthew Paul Thomas wrote on 06/08/12 12:04:
> ...
> 
> An average of 1.4 crashes/user/calendar-day is far too high. 
> Suppressing the error messages won't fix that. Only fixing the 
> errors will.
> 
> ...

Correction: Steve Langasek's comment that "the value on this graph
will always be >= 1" has helped us realize that we're doing the
calculation wrong. We're dividing the number of error reports by the
number of machines reporting that day. What we actually want is to
divide by the *total* number of machines from which their users would
typically send error reports.

We can't know that exactly, but we can get much closer by using the
number of machines that have sent any error reports in the past 90
days.  That won't require any
client-side changes.

It's likely that will result in the average going below 1.0. So the
"ten error prompts per week" is almost certainly an overstatement.

- -- 
mpt
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAlAg7rcACgkQ6PUxNfU6ecrNpACcCWXPamwJmICDuAeBWNPzWESm
N0sAoKNxR9kOomkwmIfZ9xM0n3T/QY6+
=n455
-END PGP SIGNATURE-

-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Matthew Paul Thomas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Sebastien Bacher wrote on 06/08/12 17:27:
> ...
> 
> Let me change the angle of my suggestion, and say "we can't keep 
> the LTS with that number of error prompts", that's my position.
> 
> ...

In an ideal world, there would be no alert boxes of any sort. I look
forward to the time when people using Ubuntu can go for weeks without
seeing an error message, whether from whoopsie or anything else.

But optimizing purely for the number of error prompts is the wrong
goal. The situations we're discussing are situations where *something
has already gone wrong*. We then have a choice between explaining what
went wrong, or leaving it a mystery.

It may be hard for Ubuntu contributors to appreciate the need to
explain what just went wrong, because Ubuntu contributors are
self-selected to be familiar with the situation where errors go
unexplained. And so instead you focus on things like how many people
will be working on fixing errors in 12.04.1. But that isn't actually
relevant to the choice of whether to explain errors to users or not.

Now, you propose that so few of the crashes in Ubuntu 12.04.1 need to
be explained, that *none* of them should be. But you haven't provided
any data to support this. The claim that most crashes are in services
isn't supported by the evidence. (Of the most common reported errors,
yesterday 17/50 were; today only 7/50 were.)

If we can isolate particular classes of error that we can be confident
don't need explaining, then great! Let's not show an error prompt for
those. (We could batch up the error reports, and send them next time
an error occurs that does need to be explained.) For example, perhaps
you could describe how to identify errors that happened at logout or
shutdown.

Cheers
- -- 
mpt
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAlAg5DAACgkQ6PUxNfU6ecrODwCeOqOXerbHLaPrwYHb3OYggM7U
CO4AoNJwJ9RoIn/SAS5StTYnTv9JIW/c
=fMB/
-END PGP SIGNATURE-

-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Oliver Grawert]

hi,
Am Dienstag, den 07.08.2012, 08:19 +0200 schrieb Didier Roche:

> - the most important point in my opinion is that we are making the user 
> paying the price about the error. We are interrupting them when they are 
> using application X (or even maybe when they are watching a video on 
> youtube/totem) and popping up about "hey, the service Y crashed, do you 
> want to report it?". I guess this is my major issue about the problem 
> and really expecting your work on batching the error reports in a 
> sensible way to fix it. 

while i dont agree that switching off whoopsie is a good idea (instead
we should give the teams more resources or even hire a complete LTS
maintenance team at some point) i totally agree with the above ...

the problem with whoopsie is for me as a user less the frequency than
the behavior:

- popping up for issues that didnt happen in the current session should
completely be avoided (yes, we might miss some shutdown issues due to
that but it would feel way less bad in the user side if you dont have
five popups in a row for piled up stuff right after login)

- the popup completely steals my focus while i'm in the middle of typing
(or even worse when i'm at the end of a sentence and hit enter which
triggers the selected default button in the UI)

- if i dont want to care for it right now and click a different window
into the foreground there is no way at all to get back to it, it doesnt
show up in the alt+tab list nor in the launcher, my only way to get to
it again is to minimize all open windows that cover it.

i think with these three issues resolved it would feel to me personally
a lot less intrusive.

ciao
oli


signature.asc
Description: This is a digitally signed message part
-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Reducing dialogues rate (was: Re: Disabling whoopsie by default in the 12.04.1 release)

[Sebastien Bacher]

Le 07/08/2012 08:40, Steve Langasek a écrit :

that data that aren't giving users a bad impression (if indeed that's what's
happening).  For instance, what if we were to only pop up the whoopsie
prompt for every fifth crash on a user's system on a stable release?  The
per-user dialogue rate would be slashed by 80% (er, except for the fact that
That would be good to reduce the spamming but it would also breaks what 
Evan and Matthew consider one of the main features, telling users about 
problem when they happen and gives an easy way to restart the 
applications when it closed. We could maybe do something like that on 
session start though? If you log in and 5 reports are "waiting" on the 
disk we should probably not have 5 dialogs displaying in sequence...


Cheers,
Sebastien Bacher

--
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Sebastien Bacher]

Le 07/08/2012 08:14, Steve Langasek a écrit :



If there was this much disparity between the rate of error dialogs and the
rate of SRU bug fixes, it would have been nice to make this a focus for
12.04.1 work.  But that's water under the bridge now.
It has been defined as a focus and several people tried to keep a 
dynamic around fixing bugs from the list (thanks mvo, the s-c team for 
their efforts in that regard), desktop has been reviewing the list 
regularly and dispatching most important bugs as resources permitted, 
the issue is that the resources are limited and we didn't make the 
progresses we wanted.





errors.ubuntu.com currently shows an average number of crashes per day of
1.39, which I agree is unreasonably high.  But this number doesn't reflect:

  - users who experienced no crashes
  - users who experienced crashes that they chose not to report
- the fact that apport stop nagging about,reporting issues after 3 
instances to not "spam" the user with dialogs, which in practice means 
the datas become less useful on installed systems over time and the most 
frequent issues stop being reported by existing users (they still come 
through from new installs though)




I don't know what sources you're alleging are unmaintained here.  If these
are packages that are contributing substantially to the daily error rate
across all our entire userbase, then they're almost all going to be packages
in main, and those are not unmaintained.  If the level of maintenance hasn't
led to an adequate reduction in the number of python exceptions, I think
that's because this hasn't been made a priority - and that doesn't surprise
me since this wasn't brought up as a problem before now.
Taking the list for this month and bugs with at least 10 reports a day, 
which makes 285 entries, the packages with at least 5 lines in that set are:
(the numbers are the numbers of different issues by source, not the 
number of reports, e.g: jockey has 24 different bugs on the top list)

129 software-center <- actively maintained
 57 update-manager <- not unmaintained but got 6 bugs fixed since 
precise in SRUs

 24 jockey <- 0 SRU
 18 unity
 18 aptdaemon <- 1 SRU, fixing 4 bugs since precise
 16 software-properties <- 1 SRU, 1 bug fixed since precise
 12 sessioninstaller <- 1 SRU, 1 bug fixed since precise
 12 gwibber
 12 apport
 10 unity-lens-video
 10 rhythmbox
  9 compiz
  8 update-notifier
  8 ubuntuone-client
  8 oneconf
  8 emesene
  8 blueman <- unmaintained, has one of the most frequent error 
unaddressed

  7 indiv-screenlets
  6 vlc
  6 ubuntu-sso-client
  6 colord
  5 apt-xapian-index

I don't mean to pick on anyone there by looking at those I pointed:

- update-manager is unmaintained by the Ubuntu teams, mvo as the 
historical maintainer is fixing some issues on best efforts but clearly 
doesn't have the time to look at all those issues
- jockey is virtually unmaintained, deprecated in quantal and pitti with 
his role change didn't get much of a chance to SRU it
- aptdaemon is basically in the same situation than update-manager, 
glatzor is doing good job working on it but nobody has been taking up on 
SRUing fixes
- software-properties is virtually unmaintained (the 1 bug fixed is 
because we tried to dispatch some of the issues from the list in the 
desktop team and Didier fixed one)
- sessioninstaller is the same story than software-properties, 
unmaintained, mterry fixed one issue from the top list but that's about it

- update-notifier, same...
- apt-xapian-index, same...

You can claim that those are maintained but in practice that would be 
misleading, from the list it seems like having mvo moving to new 
challenges is what hurts us the most here, we should probably discuss 
how to better maintain our "package management tools" set




Without committing to any particular solution yet, I think we need to first
come up with an objective metric of what we think is an acceptable user
experience, then figure out what it takes to achieve that, and decide if we
think that's feasible.


Turning whoopsie off for precise leaves users without an explanation
when any kind of application failure occurs. It means that we cannot
measure stability of 12.10 against 12.04. It completely leaves us in
the dark on the state of 12.04 after the .1 release. It's really going
at the messenger with a hatchet.

That's what we have been doing for 8 years, it's not perfect but
it's not the end of the world, I guess we could do it for another
release?

What would prevent this same argument from being made in another 6 months'
time about 12.10?  Furthermore, 12.04 being an LTS, it *is* supposed to
remain a target of bugfixing for some time to come; if errors.ubuntu.com is
providing important information about what our bugfixing priorities should
be, then that's true for LTS SRUs as much as it is elsewhere, and we should
carefully weigh the loss of that 

Re: Disabling whoopsie by default in the 12.04.1 release

[Steve Langasek]

On Mon, Aug 06, 2012 at 01:09:54PM +0200, Sebastien Bacher wrote:
> >I think we could argue about whether it's showing up "too often." It's
> >showing up precisely as often as the user is experiencing crashes. At
> >present, this is 1.47 times a day on average (a value we wont know if
> >we turn off error reporting). If consensus is that's too often, then
> >we should be focused on bringing that number down by fixing the most
> >pressing issues.

> That's 10 times a week, that seems very often to me yes.

My understanding is that this is not a correct extrapolation of the numbers
in the graph - that the average given for each day is based on the number of
users who sent submissions *on that day*, not on the number of users who
have ever submitted.  So the value on this graph will *always* be >= 1, and
that doesn't actually tell us anything AFAICS.  Hopefully Evan can provide
some more data here.

> >If I was still working on the installer, I could go to the website
> >right now, punch ubiquity into the box and instantly have a list of
> >what I need to focus my attention on. I *wish* I had this five years
> >ago when Launchpad already had thousands of open bugs for ubiquity.
> Right, that's precisely the issue ... we collect those datas but for
> who? Out of software-center who has dedicated people I would say
> there has been very little progress on the most commonly reported
> bugs since precise.

I feel I need to set the record straight here.  The top five crashes on
errors.u.c on Foundations team packages for the month of August are:

  bug #1027648 - ubiquity
  bug #818760  - update-manager
  bug #926340  - aptdaemon
  bug #972436  - update-manager
  bug #628104  - update-manager

Of these:

 - 1 has just been published as an SRU (so will probably fall down the list
   soon)
 - 1 is in progress and still targeted to 12.04.1
 - 1 has been accepted into precise-proposed last week, and errors.u.c
   confirms that no one is reporting the crash in the new version
 - 1 has been triaged for a while and will be one to look at after 12.04.1
 - 1 is a surprise recent addition to the priority bug list, noticed by way
   of errors.u.c.

This is in addition to the other top crashers from errors.u.c that have
already been fixed since the precise release - there have been a number of
them, but I can't tell you how many because we weren't keeping score.

So, no, I don't agree that there has been little progress outside of
software-center.  Foundations is absolutely making use of this to help us
prioritize bugfixing.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: Digital signature
-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Steve Langasek]

On Mon, Aug 06, 2012 at 12:05:58PM +0100, Evan Dandrea wrote:

> I don't think we can claim having insufficient resources to fix every
> issue as a defense for not wanting to know about the most pressing issues. 
> Quite the opposite, actually.  If we have fewer developers working on
> this, then it's all the more critical that they work on the problems that
> affect the most users.

I don't think Seb means that he doesn't want to know about the pressing
issues, only that the way we're finding out about these issues is doing more
harm to the user experience than good.  I don't necessarily agree, but I
think it bears looking at.

I also think we should think outside the box when looking for ways to get
that data that aren't giving users a bad impression (if indeed that's what's
happening).  For instance, what if we were to only pop up the whoopsie
prompt for every fifth crash on a user's system on a stable release?  The
per-user dialogue rate would be slashed by 80% (er, except for the fact that
our definition of "crashes per user per day" on errors.ubuntu.com can never
drop below 1), but the *relative* frequency of crash reports should stay
approximately the same - statistically speaking.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: Digital signature
-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Didier Roche]

Le 06/08/2012 19:49, Dylan McCall a écrit :

On Mon, Aug 6, 2012 at 6:20 AM, Matthew Paul Thomas  wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Sebastien Bacher wrote on 06/08/12 12:48:

Le 06/08/2012 13:04, Matthew Paul Thomas a écrit :

- It makes relaunching a crashed application much easier.

Right, most of the issues we get are with services and not
applications though

That isn't true, unless today is a freak exception. Right now, out of
the 50 most common errors, only 17 are from services. The rest are
from applications.

Isn't that _reported_ errors? Do you have any numbers for error popups
that have been dismissed? Personally, I almost always dismiss the
system error popups. They are vaguely worded and usually for the same
problem in mission-control. The application errors, on the other hand,
are upfront about what is broken. It's likely I have just seen that
application crashing. I know (and care about) whatever is going on.
Oh, it also helps that lots of background stuff loves to crash during
shutdown / suspend / resume (resulting in crash popups when I log in),
while application crash popups are at slightly less annoying, and more
meaningful, moments. I'm willing to accept that I could be an
exception, but I suspect the numbers of reported errors might be
biased in this way.

I don't run a computer lab, but I did upgrade someone's computer to
Ubuntu 12.04. A few days later, I felt like a total jerk as I stepped
him through disabling error popups using terminal commands. (I think
he opened xterm instead of gnome-terminal, too). After that, he has
been very happy with the system. It isn't that he doesn't like
helping: he just doesn't want to be bothered about it, at random, by
some popup that reads like the sky is falling. He has work to do, and
he likes to focus on it rather than his computer. That's why he
switched to Ubuntu a few years ago.

That's exactly a similar feedback that I received from the french 
community. I was first surprised when, shortly after the release, people 
mentioned precise as a "disaster", the "less stable release we ever 
had", or again "shameful LTS" (vaguely translated from the French 
forum). I also saw people reinstall back 11.10.


I only understand at a booth when people complained about the crash 
dialog report and that it was the cause of this low quality perception. 
Multiple cause on that:
- the crash is nagging them everyday, multiple times a day. "Why again 
do you nag me about apps  crashing even if I reported it three 
times?". We are basically shooting in our feets and until we are ready 
to deal with every crash report (and it was useful at the very beginning 
of the released version), now we are just about to continue bothering 
the user without having the man power to fix issues behind.
- then, most of the dialog seems to be dismissed, so as Dylan was 
telling, this is not completely a good metric and only showing a small 
part of the reality. With time, people are reporting less and less, so 
even if we saw less crash reports on whoopsie for the LTS, we can't go 
to any conclusion.
- some people think that whoopsie/apport is buggy. Indeed, they saw the 
dialog about software-center crashing and still see the ui (even 
updating). So "no, it didn't crash, I won't report that crash". I will 
concure what was already told on the thread: this is really useful and 
important to give feedback when something happens (like an X application 
crashing) and telling "sorry" as well as giving them a way to restart 
the application, however, there are too many cases when it seems to only 
impact a background service, and even if it impacts the ui (like you 
have to click again a button to get your results as the service 
interrupted), I don't think that users stopped on that. They would 
prefer clicking again than reporting a crash in addition to this click. 
Also note that the crash dialog doesn't give any clue on how to get to 
the wanted action. It just adds one more step from the user perspective 
(even if it reports us useful data) with no obvious clue for them.
- the most important point in my opinion is that we are making the user 
paying the price about the error. We are interrupting them when they are 
using application X (or even maybe when they are watching a video on 
youtube/totem) and popping up about "hey, the service Y crashed, do you 
want to report it?". I guess this is my major issue about the problem 
and really expecting your work on batching the error reports in a 
sensible way to fix it. However until it's there and is SRUed to the 
LTS, I'm really in favor of turning whoopsie off by default in the LTS.


Thanks,
Didier

--
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Steve Langasek]

Hi Seb,

On Mon, Aug 06, 2012 at 06:27:52PM +0200, Sebastien Bacher wrote:
> Let me change the angle of my suggestion, and say "we can't keep the
> LTS with that number of error prompts", that's my position.

> The possible way out listed so far:
> - turn off whoopsie
> - increase our involvement on bugs fixing
> - drop buggy softwares (not really practical in the LTS, I doubt we
> can drop update-manager or jockey like that)
> - other idea you got

> We need to pick one and make it happen, i.e we can't say "we should
> fix the issues" and not to do, so I would suggest that we either
> - allocate extra resources and focus to fix the issues on the LTS
> or
> - turn off apport

> (I don't think dropping software is practically possible)

Thanks for this reframing of the question, I think that's very helpful.

In an earlier message, Martin spoke of whoopsie being a "great experiment".
Let me say that this was never envisioned as an experiment; the intent here,
as discussed at UDS-P, was to provide a solid and low-friction error
reporting system that could be kept enabled for stable releases /over the
long term/; it was a core design goal to capture information about the
crashes affecting users running the stable releases, so that we could
properly prioritize our work around SRUs.  If that's not what's happening, I
certainly agree that we need to address this.

If there was this much disparity between the rate of error dialogs and the
rate of SRU bug fixes, it would have been nice to make this a focus for
12.04.1 work.  But that's water under the bridge now.

However, I have a hard time judging what the correct solution here is
because the information available remains rather incomplete.  I have not
heard the same feedback that you have about precise being buggy, and I'm not
sure that's actually representative of users' experience or if this is a
self-selecting sample - and if the latter, whether there are other common
factors that make the experience much worse for these users.  It would be
incredibly ironic if we turned from the path of acquiring a data-driven
understanding of Ubuntu quality solely in response to anecdotal feedback!

errors.ubuntu.com currently shows an average number of crashes per day of
1.39, which I agree is unreasonably high.  But this number doesn't reflect:

 - users who experienced no crashes
 - users who experienced crashes that they chose not to report

In fact, it doesn't even tell us how many users are included in the set
being averaged, because it only shows the top crashes.  Are there 500 more
crashes per day that get cut off the bottom of the report? 5000? 500,000?

Evan, can you provide some clarity here?  How many users is this average
taken from?  What's the standard deviation on the number of crashes per day? 
What's the turnover of users reporting crashes from day to day?  (I.e., it's
quite different if the same 5000 users are reporting 1.4 crashes per day,
than if 5000 *unique* users are reporting 1.4 crashes each day.) These are
the kinds of information I think we need to have in order to make an
informed decision.

If we can't quickly answer these questions, I think that's a strong argument
for turning whoopsie off *until we can*, with the intent of re-enabling it
once we're sure that we're on track.  Yes, that would be a setback for
gathering data about the most severe bugs; but the doubt here is about what
it's costing us (in terms of user experience) to gather that data and
whether we're making effective use of it in practice, and that's something
it's really important for us to know.  Just as long as we bear in mind that
the goal is to work out the kinks and get it re-enabled - among other
things, whoopsie is crucial to us improving our SRU process, and that
definitely requires it to be on in stable releases.

> >I don't see how uncaught Python exceptions are harmless. Even if they
> >were, codifying this belief of harmlessness in a try, except, pass
> >block is a very small patch.

> It is, yet many of those sources are unmaintained and that didn't
> happen in the 3 months between 12.04 and 12.04.1, we need to catch
> up with reality and admit that we don't keep up with issues.

I don't know what sources you're alleging are unmaintained here.  If these
are packages that are contributing substantially to the daily error rate
across all our entire userbase, then they're almost all going to be packages
in main, and those are not unmaintained.  If the level of maintenance hasn't
led to an adequate reduction in the number of python exceptions, I think
that's because this hasn't been made a priority - and that doesn't surprise
me since this wasn't brought up as a problem before now.

Without committing to any particular solution yet, I think we need to first
come up with an objective metric of what we think is an acceptable user
experience, then figure out what it takes to achieve that, and decide if we
think that's feasible.

> >Turning whoopsie off for precise leaves 

Re: Disabling whoopsie by default in the 12.04.1 release

[Dylan McCall]

On Mon, Aug 6, 2012 at 6:20 AM, Matthew Paul Thomas  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Sebastien Bacher wrote on 06/08/12 12:48:
>>
>> Le 06/08/2012 13:04, Matthew Paul Thomas a écrit :
>>>
>>> - It makes relaunching a crashed application much easier.
>>
>> Right, most of the issues we get are with services and not
>> applications though
>
> That isn't true, unless today is a freak exception. Right now, out of
> the 50 most common errors, only 17 are from services. The rest are
> from applications.

Isn't that _reported_ errors? Do you have any numbers for error popups
that have been dismissed? Personally, I almost always dismiss the
system error popups. They are vaguely worded and usually for the same
problem in mission-control. The application errors, on the other hand,
are upfront about what is broken. It's likely I have just seen that
application crashing. I know (and care about) whatever is going on.
Oh, it also helps that lots of background stuff loves to crash during
shutdown / suspend / resume (resulting in crash popups when I log in),
while application crash popups are at slightly less annoying, and more
meaningful, moments. I'm willing to accept that I could be an
exception, but I suspect the numbers of reported errors might be
biased in this way.

I don't run a computer lab, but I did upgrade someone's computer to
Ubuntu 12.04. A few days later, I felt like a total jerk as I stepped
him through disabling error popups using terminal commands. (I think
he opened xterm instead of gnome-terminal, too). After that, he has
been very happy with the system. It isn't that he doesn't like
helping: he just doesn't want to be bothered about it, at random, by
some popup that reads like the sky is falling. He has work to do, and
he likes to focus on it rather than his computer. That's why he
switched to Ubuntu a few years ago.

--
Dylan

PS: For what it's worth, Microsoft's Action Centre thing from Windows
7 might actually be an interesting model for this. It is mostly a
useless thing that nags people to set up backups, but it also has a
nice bit that lists collected system errors. It's useful to actually
diagnose problems that are occurring. In Ubuntu all we seem to do is
nag the user to report errors (which are promptly forgotten), and
there's no real sense of getting something back for the trouble. What
if that data was aggregated locally, too, so a user could see that a
particular component is crashing really frequently? Or report issues
from crash popups that had been previously dismissed? It might make
the whole thing more agreeable ;)

-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Sebastien Bacher]

(Trying to shorter the response a bit so the email keep being readeable)


Sure, but as Matthew points out, these dialogs serve two purposes:
1. To let the user know what just happened.
2. To let us prioritize work effectively.

They do not make promises about work getting done. If there's software
that is used by a sufficient number of people to move the needle on
the front page of http://errors.ubuntu.com but we do not have the
resources to fix the bugs in it, then we should have conversations
about dropping that application as a means of reducing the number or
frequency of errors.
Your reply are mostly around the line of "we should fix issues", which 
is fine but doesn't address the concerns there.


Let me change the angle of my suggestion, and say "we can't keep the LTS 
with that number of error prompts", that's my position.


The possible way out listed so far:
- turn off whoopsie
- increase our involvement on bugs fixing
- drop buggy softwares (not really practical in the LTS, I doubt we can 
drop update-manager or jockey like that)

- other idea you got

We need to pick one and make it happen, i.e we can't say "we should fix 
the issues" and not to do, so I would suggest that we either

- allocate extra resources and focus to fix the issues on the LTS
or
- turn off apport

(I don't think dropping software is practically possible)

If we go for "we allocate extra resources" I would like to see a 
concrete plan for making it happen with an commitment on results (taking 
the daily count by user < 1 for example) and revisiting the decision 
later in the cycle (before LTS .2).



I don't see how uncaught Python exceptions are harmless. Even if they
were, codifying this belief of harmlessness in a try, except, pass
block is a very small patch.
It is, yet many of those sources are unmaintained and that didn't happen 
in the 3 months between 12.04 and 12.04.1, we need to catch up with 
reality and admit that we don't keep up with issues.



Can you elaborate on what's causing these applications to fail on log
out? I don't see how logging out by itself invalidates the contents of
an error report.

Usually at some point xorg closes and the dbus session bus goes away 
which leads applications and services which were still running and not 
stopped to hit an error while trying to access dbus or the xserver which 
went away, since usually that happens after the session closed it's 
mostly noise (it wouldn't happen if we had a cgroup or something which 
would ensure all processes are closed when the session goes away but 
that's not the case at the moment)



Even if
no one is going to do anything about it, any regression should be at
least acknowledged and its impact known.

That's a low probability benefit:
- not enough users run proposed to get those issues showing up on the 
report before they move to updates (at which point we already screwed up 
and let a regression in -updates which shouldn't happen)

- they need to rank enough so we notice them on the report
- we would need a way to spot out "new issues", the current "by 
frequency" ranking doesn't tell us what issue started in a recent upload

Turning whoopsie off for precise leaves users without an explanation
when any kind of application failure occurs. It means that we cannot
measure stability of 12.10 against 12.04. It completely leaves us in
the dark on the state of 12.04 after the .1 release. It's really going
at the messenger with a hatchet.
That's what we have been doing for 8 years, it's not perfect but it's 
not the end of the world, I guess we could do it for another release?

- the evolution of this number?

I'm not sure what you mean by this. We have a graph on the front page
that shows the average number of errors per day. In the aforementioned
RT it's separated out into lines for each Ubuntu version.
The current graph seems to indicate we rank around 1.45 
issue/day/submitter, it only goes back to 07/31/12 though (or it would 
maybe go back further if selecting "year" in the combo was working but 
it seems to timeout, it returns an "An error occurred while trying to 
load the most common problems." after a while on "Loading" here) ... 
what was the number at precise release time. Can we see how much 
progress we did and the look of the curve?


Yes, in the aforementioned RT, you see a graph on the top of the
problem page with the instance count over time. You also get the
version-by-version breakdown on the problem page that I mentioned
above.
Ok, I guess I need to wait on the RT to be closed and to see the update 
to see what's changing exactly but that seems good progresses, thanks 
for that

I suspect they care more about the errors that happen than the dialogs
that they get for them.

It's worth noting since you've raised Windows that other mainstream
operating systems do exactly what we're doing here. When applications
crash on Windows or OSX, you get a similar dialog.
Right, those don't have issues happening 10 times a week though,

Re: Disabling whoopsie by default in the 12.04.1 release

[Jeremy Bicha]

On 2 August 2012 17:31, Sebastien Bacher  wrote:
> What do people think about turning whoopsie off for 12.04.1?

I strongly support turning off the error message pop-ups for Ubuntu
12.04. I run a lab where we regularly get visitors who have never
knowingly used Linux on a PC before. While the popup is supposed to
eventually make Ubuntu less buggy, it feels like Ubuntu is more buggy.
Therefore I've disabled apport since upgrading the lab to 12.04.

I was surprised when I realized that it was not considered a bug for
these error reports to be kept turned on once the release was final. I
also was confused that System Settings>Privacy>Diagnostics>Send error
reports didn't actually stop the popups. I think deploying this
post-release for the first time in an LTS will cause many average
users to believe that 12.04 LTS is not very high quality and not ready
for use.

We want people to use Ubuntu 12.04 LTS. Constantly reminding users
that 12.04 is a broken buggy operating system will likely push them to
use something else. Something else could be 11.10, a different Linux
distribution, or Windows. This is discouraging to those who put a lot
of time and work into fixing as many bugs as possible for this
release.

I agree that we should keep error reporting on for 12.10 and future
releases. I expect that the popups will be quite a bit less
frequent/intrusive for 14.04 LTS.

Jeremy

-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Evan Dandrea]

On Mon, Aug 6, 2012 at 12:09 PM, Sebastien Bacher  wrote:
> Hey Evan, thanks for the reply!

Sure thing :)

> That's 10 times a week, that seems very often to me yes. I would be all for
> fixing the issues but in reality the resources allocated to that don't allow
> to make strong enough progress (out of software-center that you pointer
> before which is lucky to have a full team dedicated to it)

Sure, but as Matthew points out, these dialogs serve two purposes:
1. To let the user know what just happened.
2. To let us prioritize work effectively.

They do not make promises about work getting done. If there's software
that is used by a sufficient number of people to move the needle on
the front page of http://errors.ubuntu.com but we do not have the
resources to fix the bugs in it, then we should have conversations
about dropping that application as a means of reducing the number or
frequency of errors.

>> https://wiki.ubuntu.com/ErrorTracker#When_there_are_multiple_simultaneous_errors
>
> Is there any chance that would land into the LTS?

I'll push it to the top of my list and try to get it done and tested
by the 16th.

>> If we disable reporting for these issues, then we'll never know the
>> frequency at which they're occurring or what's causing them. We'll be
>> right back at developers having to chose between failing hard to
>> surface an issue or partially recovering from it and never knowing its
>> true extent or the damage it inflicts.
>
> Well, by reviewing the errors since precise a good part of the issues have
> been judged harmless for users (often untrapped python exceptions, issues
> happening at session logout or random glitches from services like oneconf
> which do actions on regular basis and where missing a run has no real
> effect). Consensus in the desktop team is that while those are bugs they are
> often worth not bothering users about and create a sentiment of instability
> where not needed.

I don't see how uncaught Python exceptions are harmless. Even if they
were, codifying this belief of harmlessness in a try, except, pass
block is a very small patch.

Can you elaborate on what's causing these applications to fail on log
out? I don't see how logging out by itself invalidates the contents of
an error report.

> What I meant there is not that the datas are not useful, is that we are
> collecting datas about versions we will not work on (because of the
> resources allocated to the stable release mostly).
> Also the suggestion is to turn whoopsie off only for one extra release
> (precise) and to keep it on on other series.

Well, we will be working on these releases. There are four point
releases scheduled for Ubuntu 12.04. The engineering staff assigned to
these releases may grow smaller, but that will not prevent developers
from uploading software to 12.04. If they do and that software causes
failures that did not exist before, I want them to know about it
quickly and I want the release team to know about it quickly. Even if
no one is going to do anything about it, any regression should be at
least acknowledged and its impact known.

Turning whoopsie off for precise leaves users without an explanation
when any kind of application failure occurs. It means that we cannot
measure stability of 12.10 against 12.04. It completely leaves us in
the dark on the state of 12.04 after the .1 release. It's really going
at the messenger with a hatchet.

> Again the issue are:
> - we don't have enough people dedicated to work on the precise issues, we
> collect datas but don't make use of them by allocating resources to fix the
> issues

We'll have people uploading to precise-updates long after 12.04.1 is
out the door. We should be monitoring the impact of this. We will have
problems that surface only after this time.

> - it's getting hard to use the current errors.ubuntu.com summary because you
> can't filter out things which got a fixed rolled out from those which don't,
> half of "the main page" are probably issues we tackled but where the fix
> didn't reach users, as somebody planned work I would like those out of the
> overview because there is not a lot my team can do there, but it doesn't
> prevent us to see what are the real remaining issues.

This is a problem of our updates mechanism more than anything else.
The numbers are real. We just do a very poor job of getting fixes in
the hands of users.

As mentioned, as soon as the RT 54702 lands, you will be able to see
the version-by-version breakdown of an individual problem. This will
happen soon. We're just working through the dependency chain for
python-tastypie in lucid with builds ongoing as I type this email.

I also have the aforementioned branch to grey out issues that have not
occurred yet in the most recent published version.

>> If I was still working on the installer, I could go to the website
>> right now, punch ubiquity into the box and instantly have a list of
>> what I need to focus my attention on. I *wish* I had this five

Re: Disabling whoopsie by default in the 12.04.1 release

[Evan Dandrea]

> Hello all,
>
> Sebastien Bacher [2012-08-02 23:31 +0200]:
>> I know that most of the cons are addressable but until we do address
>> them the consensus form the people I talked to seems to be that the
>> cost-benefit is largely not in our favour at this point so I would
>> recommend we do disable it by default for 12.04.1
>
> Big +1 from me. It has been a great experiment, advancement, and also
> help for 12.04 so far, and did improve the quality. However, after
> 12.04.1 there will be much less focus on precise as we will not have a
> dedicated team for it any more, and the worst issues should have been
> shaken out now. Having fewer people working on it makes the
> cost-benefit ratio even worse.

Ubuntu 12.04 is not static. This discussion revolves around a point release -
a large number of changes occurring to this already-released operating system.
Indeed, there will be many changes still via -updates and future point
releases.

These all bring with them their share of bugs. How will we know the extent of
them without http://errors.ubuntu.com ?

I don't think we can claim having insufficient resources to fix every issue as
a defense for not wanting to know about the most pressing issues. Quite the
opposite, actually. If we have fewer developers working on this, then it's all
the more critical that they work on the problems that affect the most users.

As I'm sure you recall, Matthew designed the interface to deemphasize a
connection between the user and someone fixing the issue. There's no link to a
bug report or problem page that they can follow along with. There's no promise
that the issue will be resolved. So having less engineers working on 12.04
wont change their perception.

> Apport as it was on precise (and still is in Quantal) has not really
> been designed for usage in a stable release. For example, the rate
> limitation should be a lot more aggressive in stables, and we need to
> do something about the presentation of crashes that are not obviously
> connected to the UI, such as crashes in threads and respawned
> services.

Just echoing Matthew's comment that we should turn off rate-limiting when not
reporting to Launchpad. I've filed a bug for this so we can debate the merits
outside this thread:

https://bugs.launchpad.net/ubuntu/+source/apport/+bug/1033471

> Absolutely! Thanks Evan for all this, it's great to see this evolve so
> systematically.

Thanks

-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Evan Dandrea]

Hello everyone. Apologies for the late reply; I was on holiday all
last week. Please CC myself and Matthew on replies to this thread as
we are not subscribed to the list.

Thanks for taking the time to raise this, Seb.

On Thu, Aug 2, 2012 at 10:31 PM, Sebastien Bacher  wrote:
> That's something quite some people raised as an issue since precise, the
> frequent whoopsie dialogs in the LTS gives users the feeling that precise is
> unstable (it seems often qualified to buggier over previous release for no
> reason out of the number of error dialogs showing up to report bugs).
>
> I've discussed the issue with different people in different teams, here is
> my try to a summary of the pro and con:
>
> Pro:
> - it gives us infos on what issues users run into
> - it gives feedback to users on what happen when a program close while they
> are using it
>
> Con:
> - it's showing up too often and giving the impression to users that the
> system is buggy

I think we could argue about whether it's showing up "too often." It's
showing up precisely as often as the user is experiencing crashes. At
present, this is 1.47 times a day on average (a value we wont know if
we turn off error reporting). If consensus is that's too often, then
we should be focused on bringing that number down by fixing the most
pressing issues.

Matthew has graciously tackled reducing the number of dialogs on the
screen at any one time, a longstanding issue with apport that predates
the error tracker. In the future, multiple errors that occur around
the same time will be grouped into a single dialog:

https://wiki.ubuntu.com/ErrorTracker#When_there_are_multiple_simultaneous_errors

If getting this into Ubuntu is a top priority for you, let me know and
I'll raise it on my list.

> - it's often showing programming errors which don't impact users (untracked
> exception from python softwares or services by example), what users get the
> most notified about is glitchs about things like ubuntuone services,
> oneconf, software-center ... most of those are not user visible issues and
> the frontend would handle the glitches fine without whoopsie notifying the
> users

I disagree; these very well may impact users. There is simply no way
of knowing whether a background service crashing affects the task the
user is currently undertaking.

If a DBus service crashes and is automatically restarted, what happens
to the application that was partially through a call to that service?
What does the graphical interface do?

If we disable reporting for these issues, then we'll never know the
frequency at which they're occurring or what's causing them. We'll be
right back at developers having to chose between failing hard to
surface an issue or partially recovering from it and never knowing its
true extent or the damage it inflicts.

> - errors.ubuntu.com is not good enough yet that we can properly tackle those
> issues

I would ask that you elaborate on this point.

> - we don't have the resources to get where we want to be in a short
> timeframe, we are working on it but meanwhile we are impacting users for
> things we don't make real use for

This is patently false. There are things we want the website to do in
the future, sure. That does not mean we should stop collecting data.

Nearly all of the changes that affect the website happen in either the
daisy codebase (the system that receives the crashes) or the errors
codebase (the website itself).

We are taking a big data approach to this problem and collecting more
information than we are using at present. Thus, the changes we make
are mostly ways to better map the flow of information to our database,
Cassandra. It is often possible to back-populate the new structure
with the existing data.

So when a new feature to the website lands, it will immediately begin
working for releases that are already very frozen, like 12.04. This
was exactly the case with the per-version breakdown on individual
problem pages.

I challenge the notion that the website is not usable yet. Developers
are using it and there's already a wealth of valuable information on
it that flat out did not exist before. Where are you going to get a
more accurate prioritized list of errors? Where are you going to find
a version by version breakdown of where that error manifests? Where
are you going to find out how often Ubuntu is crashing for end-users
(not just developers) relative to the previous version? Where are you
going to find the problems that do not appear in Launchpad bugs?

If I was still working on the installer, I could go to the website
right now, punch ubiquity into the box and instantly have a list of
what I need to focus my attention on. I *wish* I had this five years
ago when Launchpad already had thousands of open bugs for ubiquity.

Talk to David Pitkin and the software center team. It is my
understanding that they use this website extensively to prioritize
work and fix major issues. I have constantly seen peaks and troughs as
program

Re: Disabling whoopsie by default in the 12.04.1 release

[Stéphane Graber]

On 08/02/2012 05:31 PM, Sebastien Bacher wrote:
> Hey,
> 
> That's something quite some people raised as an issue since precise, the
> frequent whoopsie dialogs in the LTS gives users the feeling that
> precise is unstable (it seems often qualified to buggier over previous
> release for no reason out of the number of error dialogs showing up to
> report bugs).
> 
> I've discussed the issue with different people in different teams, here
> is my try to a summary of the pro and con:
> 
> Pro:
> - it gives us infos on what issues users run into
> - it gives feedback to users on what happen when a program close while
> they are using it
> 
> Con:
> - it's showing up too often and giving the impression to users that the
> system is buggy
> - it's often showing programming errors which don't impact users
> (untracked exception from python softwares or services by example), what
> users get the most notified about is glitchs about things like ubuntuone
> services, oneconf, software-center ... most of those are not user
> visible issues and the frontend would handle the glitches fine without
> whoopsie notifying the users
> - errors.ubuntu.com is not good enough yet that we can properly tackle
> those issues
> - we don't have the resources to get where we want to be in a short
> timeframe, we are working on it but meanwhile we are impacting users for
> things we don't make real use for
> 
> I know that most of the cons are addressable but until we do address
> them the consensus form the people I talked to seems to be that the
> cost-benefit is largely not in our favour at this point so I would
> recommend we do disable it by default for 12.04.1 to minimize LTS
> annoyance and keep it enable from now on for new release (on the basis
> that the system will improve enough that we can catch up and that the
> perception issues is a bit less of an issue on a non LTS)
> 
> Just to give some details about the errors.ubuntu.com issues mentioned
> before (I think people are going to ask what are the problems at some
> point so I can as well reply to that here):
> - it's not possible to filter out issues which have been resolved from
> the list (so it's hard to know what has been worked on and what needs
> work still)
> - it's not possible to say what issue happens to what version and get
> stats of instances of the bugs by version, i.e to get datas on whether
> the situation improved or not for a bug
> - some of our engineers have issues login into the system to get access
> to the infos they need to work on the bugs, that situation is still not
> resolved after some months
> - other small issues, but I don't want to turn that email into a "list
> what is wrong currently", especially when we have people knowing about
> those and working on improving the situation
> 
> I would add that Evan did an amazing job so far and that
> errors.ubuntu.com has been proved very useful already (we fixed at least
> an hundred bugs, most wouldn't have show up in this way using launchpad)
> so the email is not again that work, we just feel that the current
> status of the system and the resources allocated to fixing those bugs
> gives us a situation where the benefit is not sufficient to justify the
> cost on the Ubuntu image.
> 
> What do people think about turning whoopsie off for 12.04.1?
> 
> Cheers,
> Sebastien Bacher

Thanks Sebastien for raising the issue.

My point of view on the matter is that whoopsie is a great tool, sending
us amazing data that we're only starting to see how to use.
I really think we should do anything we can to keep it running on all
releases.

Now, that being said, my point of view on the user experience is that it
sucks, sorry for being a bit harsh here, but let me tell you what
happens when I login on most of my 12.04 machines (thankfully I seem to
have the buggiest ones so it hasn't been a problem for my friends and
relatives).

 1) Boot the machine
 2) Login
 3) Get what looks like a perfectly well working desktop
 4) For the next 10 to 15 minutes, getting random apport prompts for
everything that blew up since "I don't know when (usually last reboot)"

4) is especially annoying as these are clearly old crashes that apport
for some reason didn't pick up when they happened and have been stacking
in /var/crash. I'm guessing some of these happened during session switch
or when I wasn't running a desktop environment.

In such case I'd expect apport to show me a list of everything that
crashed (all 10 of them) and let me just tick the ones I want to submit
and then be done with the lot in one go.

Instead, I get a flood of apport processes killing my CPU for 15min
randomly popping up windows, some asking for my password and some just
showing the apport window.

I see a few obvious things that might have already been reported as bugs
and that if we want to keep whoopsie enabled for 12.04, we should
probably fix (with any needed post-release UIFe be granted by the
release team if needed):

 - Either support easy b

Re: Disabling whoopsie by default in the 12.04.1 release

[Sebastien Bacher]

Le 06/08/2012 15:20, Matthew Paul Thomas a écrit :


That isn't true, unless today is a freak exception. Right now, out of
the 50 most common errors, only 17 are from services. The rest are
from applications.
Right, but if you look at e.g jockey issues some are in the backend side 
of the software, anyway let's not argue over numbers, my gut feeling 
from dealing with those bugs since precise is that about half of the 
issues deserve telling the user, the other half are harmless, session 
closing bugs, etc.


If you take a 50%-50% ratio for the sake of the argument with 10 bugs a 
week you can consider that we "spam" users 5 times a week for no good 
reason or that we avoid confusion 5 times a week for them, matter of 
perspective.


We didn't get so many complain in the past about the confusion created 
by the bugs (but maybe they just didn't reach us) but we do get quite 
some for the error dialogs...




I think that is mistaken in three ways.

Most importantly, Ubuntu 12.04.1 does not mark the end of updates to
12.04. There will be more SRUs. Hopefully some of those will make
12.04 more reliable. If we're unlucky, others may make it less
reliable. If the latter happens, how will you know?
The same way we did until precise, sure it's less ideal but as said it's 
a cost-benefit ratio and if we don't invest significant resources into 
actively fixing stable I don't see it moving enough for the datas to 
provide a benefit over what they cost us.




Second, the kind of people who install 12.04.1 (or buy a computer with
it installed) may have noticably different behavior from the kind of
people who installed 12.04 -- using different applications and
features, and encountering errors at different rates.
Right, again what's the point to know about issues and bother users if 
we do nothing about them because we don't have resources allocated to 
deal with them?



And third, there may be time-based problems that people using 12.04
can't have encountered yet (unless their clock was set wrong).
Right, it's not optimal, but we didn't have that system until precise 
and we somewhat managed to do fine, I'm sure we could deal without it 
for another release.





I didn't say it prevents you from getting error reports from 12.10. I
said it prevents you from seeing whether 12.10 is better or worse than
12.04. By "12.04" I include 12.04.1 and any later updates.


I don't think we need exact numbers to compare 12.10 to 12.04.1+, we 
know about the current state of 12.04.1 and the release team should set 
a goal, e.g "the medium submission rate should be lower than 1.3 for 
quantal". While we are doing time based released anyway those goals will 
need to be hard ones, we can't block on all parameter you need a 
variable to adjust, either let slip on goals or on time...



The main concern is that only a small fraction of those are user
visible issue.

How do you know?
I've been watching errors.ubuntu.com since precise release, dispatching 
a good number of the top issues and following up with assigned people, 
we have a good understanding of most of the bugs we fixed, when they 
happen and what's the impact.




If they are harmless, why do they need fixing at all? If they really
don't, that error message in particular could be suppressed.
They don't, that's my point, we shouldn't bother users about those, but 
our system is not smart enough at the moment to tell them apport from 
important issues.


Cheers,
Sebastien Bacher

--
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Matthew Paul Thomas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Sebastien Bacher wrote on 06/08/12 12:48:
> 
> Le 06/08/2012 13:04, Matthew Paul Thomas a écrit :
>> 
>> - It makes relaunching a crashed application much easier.
> 
> Right, most of the issues we get are with services and not 
> applications though

That isn't true, unless today is a freak exception. Right now, out of
the 50 most common errors, only 17 are from services. The rest are
from applications.

>> - From the next errors.ubuntu.com rollout, it will let release 
>> managers and other contributors see whether Ubuntu Q is more
>> reliable than 12.04.
> 
> I never suggested turning of whoopsie in Q, so we will get metrics
> in Q in any case. We have metrics from precise at release time and
> .1 time, that should be good enough to know where Q stands. I don't
> think lacking a metrics on 12.04.1 post .1 compared to Q is a big
> deal, the LTS shouldn't change much, we want to know where Q stands
> though and we will know since we will keep the system running for
> it.

I think that is mistaken in three ways.

Most importantly, Ubuntu 12.04.1 does not mark the end of updates to
12.04. There will be more SRUs. Hopefully some of those will make
12.04 more reliable. If we're unlucky, others may make it less
reliable. If the latter happens, how will you know?

Second, the kind of people who install 12.04.1 (or buy a computer with
it installed) may have noticably different behavior from the kind of
people who installed 12.04 -- using different applications and
features, and encountering errors at different rates.

And third, there may be time-based problems that people using 12.04
can't have encountered yet (unless their clock was set wrong).

>> With previous versions of Ubuntu, windows would disappear or
>> features would stop working, and people wouldn't know why. The
>> only difference is that now the failure comes with an
>> explanation. It would be best if the failure didn't occur at all,
>> but if it does, it's better for it to be explained than to be
>> mysterious.
> 
> That's again assuming that the errors we received are real bugs
> having an user visible impact, it's true in some cases but I would
> challenge that being the most frequent case, we have been looking
> at those errors for 3 months and most of them are bugs reported
> against services, or happening at session closing, or harmless
> exception uncaught in python programs.

As above, most of them are not against services. And even if they
were, so what? If Bluetooth stops working, for example (a couple of
today's top 50 are in blueman), explaining the error can help someone
understand that a Bluetooth connection failed because of a problem
with Ubuntu, not because they made a mistake. Same if sessioninstaller
fails, or aptdaemon.

>> Turning off error reporting would leave the release team
>> ignorant about whether 12.10 is an improvement on 12.04. But this
>> is not a release-team-specific question. It would make Ubuntu
>> developers in general less productive, and it would make Ubuntu
>> worse for end users.
> 
> The suggestion is to turn if off for 12.04.1, can you explain how
> it prevents us to get metrics on 12.10?

I didn't say it prevents you from getting error reports from 12.10. I
said it prevents you from seeing whether 12.10 is better or worse than
12.04. By "12.04" I include 12.04.1 and any later updates.

>> If you think there are ways the end-user presentation can be
>> improved, I'm happy to take suggestions. Routing around that by
>> asking the release team to disable it altogether is not cool.
> 
> Evan said that work was ongoing on "batching reports to submit
> several together", that will be good. I don't think there is
> magical ways around it otherwise, it's just that 10 bugs a week is
> too much errors dialog popping up.

That is shooting the messenger. Yes, ten errors a week is too many.
Suppressing the error alerts doesn't make that better, it makes it
worse. It's a choice between unreliable, and unreliable + unhelpful.

> The main concern is that only a small fraction of those are user 
> visible issue.

How do you know?

> To take an example a non marginal number of issues happen on
> session closing, they are due to the fact that our desktop session
> management is not really good. That's a known issue, will not be
> fixed in the LTS (the scope of the work is not appropriate for
> stable updates), and is harmless ... still it means lot of users
> get a "system error happened" dialog first thing after next login.
> That has a real cost in user perception, the benefit we get back is
> null though ...
> 
> ...

If they are harmless, why do they need fixing at all? If they really
don't, that error message in particular could be suppressed.

- -- 
mpt

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAlAfxKMACgkQ6PUxNfU6ecrgBwCgic/5+ZzX0R/M7UYJtfRujlq2
57IAn15H1Zf30fX92SrkgP1X6U

Re: Disabling whoopsie by default in the 12.04.1 release

[Sebastien Bacher]

Le 06/08/2012 13:04, Matthew Paul Thomas a écrit :



- - It makes relaunching a crashed application much easier.
Right, most of the issues we get are with services and not applications 
though

- - From the next errors.ubuntu.com rollout, it will let release
managers and other contributors see whether Ubuntu Q is more reliable
than 12.04.
I never suggested turning of whoopsie in Q, so we will get metrics in Q 
in any case. We have metrics from precise at release time and .1 time, 
that should be good enough to know where Q stands. I don't think lacking 
a metrics on 12.04.1 post .1 compared to Q is a big deal, the LTS 
shouldn't change much, we want to know where Q stands though and we will 
know since we will keep the system running for it.




With previous versions of Ubuntu, windows would disappear or features
would stop working, and people wouldn't know why. The only difference
is that now the failure comes with an explanation. It would be best if
the failure didn't occur at all, but if it does, it's better for it to
be explained than to be mysterious.
That's again assuming that the errors we received are real bugs having 
an user visible impact, it's true in some cases but I would challenge 
that being the most frequent case, we have been looking at those errors 
for 3 months and most of them are bugs reported against services, or 
happening at session closing, or harmless exception uncaught in python 
programs.




Turning off error reporting would leave the release team ignorant
about whether 12.10 is an improvement on 12.04. But this is not a
release-team-specific question. It would make Ubuntu developers in
general less productive, and it would make Ubuntu worse for end users.
The suggestion is to turn if off for 12.04.1, can you explain how it 
prevents us to get metrics on 12.10?


If you think there are ways the end-user presentation can be improved,
I'm happy to take suggestions. Routing around that by asking the
release team to disable it altogether is not cool.


Evan said that work was ongoing on "batching reports to submit several 
together", that will be good. I don't think there is magical ways around 
it otherwise, it's just that 10 bugs a week is too much errors dialog 
popping up. The main concern is that only a small fraction of those are 
user visible issue.


To take an example a non marginal number of issues happen on session 
closing, they are due to the fact that our desktop session management is 
not really good. That's a known issue, will not be fixed in the LTS (the 
scope of the work is not appropriate for stable updates), and is 
harmless ... still it means lot of users get a "system error happened" 
dialog first thing after next login. That has a real cost in user 
perception, the benefit we get back is null though ...



Cheers,
Sebastien Bacher

--
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Sebastien Bacher]

Le 06/08/2012 12:04, Evan Dandrea a écrit :

Hello everyone. Apologies for the late reply; I was on holiday all
last week. Please CC myself and Matthew on replies to this thread as
we are not subscribed to the list.

Thanks for taking the time to raise this, Seb.


Hey Evan, thanks for the reply!


I think we could argue about whether it's showing up "too often." It's
showing up precisely as often as the user is experiencing crashes. At
present, this is 1.47 times a day on average (a value we wont know if
we turn off error reporting). If consensus is that's too often, then
we should be focused on bringing that number down by fixing the most
pressing issues.


That's 10 times a week, that seems very often to me yes. I would be all 
for fixing the issues but in reality the resources allocated to that 
don't allow to make strong enough progress (out of software-center that 
you pointer before which is lucky to have a full team dedicated to it)




https://wiki.ubuntu.com/ErrorTracker#When_there_are_multiple_simultaneous_errors

If getting this into Ubuntu is a top priority for you, let me know and
I'll raise it on my list.

Is there any chance that would land into the LTS?



I disagree; these very well may impact users. There is simply no way
of knowing whether a background service crashing affects the task the
user is currently undertaking.

If a DBus service crashes and is automatically restarted, what happens
to the application that was partially through a call to that service?
What does the graphical interface do?

If we disable reporting for these issues, then we'll never know the
frequency at which they're occurring or what's causing them. We'll be
right back at developers having to chose between failing hard to
surface an issue or partially recovering from it and never knowing its
true extent or the damage it inflicts.
Well, by reviewing the errors since precise a good part of the issues 
have been judged harmless for users (often untrapped python exceptions, 
issues happening at session logout or random glitches from services like 
oneconf which do actions on regular basis and where missing a run has no 
real effect). Consensus in the desktop team is that while those are bugs 
they are often worth not bothering users about and create a sentiment of 
instability where not needed.



- errors.ubuntu.com is not good enough yet that we can properly tackle those
issues

I would ask that you elaborate on this point.

(I did a bit down on the original email)




- we don't have the resources to get where we want to be in a short
timeframe, we are working on it but meanwhile we are impacting users for
things we don't make real use for

This is patently false. There are things we want the website to do in
the future, sure. That does not mean we should stop collecting data.


What I meant there is not that the datas are not useful, is that we are 
collecting datas about versions we will not work on (because of the 
resources allocated to the stable release mostly).
Also the suggestion is to turn whoopsie off only for one extra release 
(precise) and to keep it on on other series.




I challenge the notion that the website is not usable yet. Developers
are using it and there's already a wealth of valuable information on
it that flat out did not exist before. Where are you going to get a
more accurate prioritized list of errors? Where are you going to find
a version by version breakdown of where that error manifests? Where
are you going to find out how often Ubuntu is crashing for end-users
(not just developers) relative to the previous version? Where are you
going to find the problems that do not appear in Launchpad bugs?
Right, I didn't say the website is totally unusable, it has proven quite 
valuable in fact.


Again the issue are:
- we don't have enough people dedicated to work on the precise issues, 
we collect datas but don't make use of them by allocating resources to 
fix the issues
- it's getting hard to use the current errors.ubuntu.com summary because 
you can't filter out things which got a fixed rolled out from those 
which don't, half of "the main page" are probably issues we tackled but 
where the fix didn't reach users, as somebody planned work I would like 
those out of the overview because there is not a lot my team can do 
there, but it doesn't prevent us to see what are the real remaining issues.

- the user perception in a lts



If I was still working on the installer, I could go to the website
right now, punch ubiquity into the box and instantly have a list of
what I need to focus my attention on. I *wish* I had this five years
ago when Launchpad already had thousands of open bugs for ubiquity.
Right, that's precisely the issue ... we collect those datas but for 
who? Out of software-center who has dedicated people I would say there 
has been very little progress on the most commonly reported bugs since 
precise.


Do we have numbers of:
- the number of issues reported by day and by use

Re: Disabling whoopsie by default in the 12.04.1 release

[Matthew Paul Thomas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Sebastien Bacher wrote on 02/08/12 22:31:
> ...
> 
> That's something quite some people raised as an issue since 
> precise, the frequent whoopsie dialogs in the LTS gives users the 
> feeling that precise is unstable

Maybe 12.04 is more stable than any previous version of Ubuntu. Maybe
it is more stable than Fedora, or Suse, or Debian. But you don't know
that, precisely because those other OSes don't have this kind of error
reporting. So how do you know that "the feeling that precise is
unstable" is not actually justified?

An average of 1.4 crashes/user/calendar-day is far too high.
Suppressing the error messages won't fix that. Only fixing the errors
will.

> (it seems often qualified to buggier over previous release for no 
> reason out of the number of error dialogs showing up to report 
> bugs).

They don't show up to report bugs. (That's not even an option, unless
you've altered crashdb.conf.) They show up to explain why something
just went wrong -- and, if it was an application crashing, to let you
relaunch it.

> I've discussed the issue with different people in different teams, 
> here is my try to a summary of the pro and con:
> 
> Pro:
> 
> - it gives us infos on what issues users run into
> 
> - it gives feedback to users on what happen when a program close 
> while they are using it

- - It makes relaunching a crashed application much easier.
- - From the next errors.ubuntu.com rollout, it will let release
managers and other contributors see whether Ubuntu Q is more reliable
than 12.04.

> Con:
> 
> - it's showing up too often and giving the impression to users that
> the system is buggy
> 
> ...

With previous versions of Ubuntu, windows would disappear or features
would stop working, and people wouldn't know why. The only difference
is that now the failure comes with an explanation. It would be best if
the failure didn't occur at all, but if it does, it's better for it to
be explained than to be mysterious.

Turning off error reporting would leave the release team ignorant
about whether 12.10 is an improvement on 12.04. But this is not a
release-team-specific question. It would make Ubuntu developers in
general less productive, and it would make Ubuntu worse for end users.

If you think there are ways the end-user presentation can be improved,
I'm happy to take suggestions. Routing around that by asking the
release team to disable it altogether is not cool.


Martin Pitt wrote on 03/08/12 03:46:
> 
> Apport as it was on precise (and still is in Quantal) has not 
> really been designed for usage in a stable release. For example, 
> the rate limitation should be a lot more aggressive in stables,

The rate limitation should be removed altogether, because it's causing
some crashes to go unexplained. (It's also skewing the statistics,
though that's less important.)

> and we need to do something about the presentation of crashes that 
> are not obviously connected to the UI, such as crashes in threads 
> and respawned services.
> 
> ...

Currently we're incorrectly showing the application crash alert for a
thread crash. The thread crash alert includes an "Ignore future
problems of this type" checkbox, which should alleviate that problem.
Hopefully this can be fixed in an SRU.

- -- 
mpt
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAlAfpMYACgkQ6PUxNfU6ecp7TACfdbHlvsfAnWIXeiHnSjB9ApiH
MVMAoKGvf+ti+P4a8EIbFiChiBUTkD5E
=VzFr
-END PGP SIGNATURE-

-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Disabling whoopsie by default in the 12.04.1 release

[Martin Pitt]

Hello all,

Sebastien Bacher [2012-08-02 23:31 +0200]:
> I know that most of the cons are addressable but until we do address
> them the consensus form the people I talked to seems to be that the
> cost-benefit is largely not in our favour at this point so I would
> recommend we do disable it by default for 12.04.1

Big +1 from me. It has been a great experiment, advancement, and also
help for 12.04 so far, and did improve the quality. However, after
12.04.1 there will be much less focus on precise as we will not have a
dedicated team for it any more, and the worst issues should have been
shaken out now. Having fewer people working on it makes the
cost-benefit ratio even worse. 

Apport as it was on precise (and still is in Quantal) has not really
been designed for usage in a stable release. For example, the rate
limitation should be a lot more aggressive in stables, and we need to
do something about the presentation of crashes that are not obviously
connected to the UI, such as crashes in threads and respawned
services.

> I would add that Evan did an amazing job so far

Absolutely! Thanks Evan for all this, it's great to see this evolve so
systematically.

Thanks,

Martin
-- 
Martin Pitt| http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)


signature.asc
Description: Digital signature
-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release