Re: yum: Critical path update in testing for 4 months?

[Michael Schwendt]

On Sun, 06 Dec 2015 20:52:19 -0800, Adam Williamson wrote:

> > > That doesn't really add up.  
> > 
> > What doesn't add up?  
> 
> We were talking about a yum update that had been sitting around for
> four months, with +10 karma.

That may be the subject, but the messages have referred to many other
updates where the maintainer could push manually. So, there have been
comments on some general problems.

> Your theory about why this kind of thing
> happens doesn't match the facts of that situation: the update submitter
> actually had to take specific action to make it so the update wouldn't
> get pushed.

How do you know? There have been Yum updates with autokarma of 15, afaik.
And then 10 would not be enough.

> If they were just 'fire'n'forget', the update would already
> have gone stable weeks ago.

You cannot tell. Updates with 0 karma could have been entered with a
threshold of 3 and would never go stable due to lack of karma.

I've entered updates with a threshold of 1 in fire'n'forget style and
neither the bug reporter(s) or any testers have voted. What a surprise!
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: yum: Critical path update in testing for 4 months?

[Kevin Fenzi]

On Mon, 7 Dec 2015 00:08:37 +0100
Reindl Harald  wrote:

> yes, i expect this when i am able to follow all fedora and 7 other 
> mailing-lists including bodhi-mails and give karma and review 100-400 
> mail samples for bayes-training every day besides my daily jobs as
> the only sysadmin for more than 30 Fedora based servers, a few
> test-servers, 4 fedora-based routers and get my *really* dayjob as
> software developer done at the same time
> 
> yes i expect that someone is able just read his mails and push a
> button on a list in a webinterface

Please keep in mind that everyone is not you. Different people have
different time commitments, priorities and levels of contribution. 

We need to make things as easy as we can for maintainers who may not
have the same time as you do. 

kevin




pgpoTaXOCGt9w.pgp
Description: OpenPGP digital signature
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: yum: Critical path update in testing for 4 months?

[Michael Schwendt]

On Mon, 07 Dec 2015 03:29 +0100, Kevin Kofler wrote:

> Adam Williamson wrote:
> > I've never understood why the idea of a 'last push of security fixes'
> > for an EOL release makes any sense. It's *EOL*. It doesn't matter if
> > there's a last-day push or not: everyone should stop using it the next
> > day, end of story. That's what EOL means.  
> 
> Realistically, some people will still be running EOL releases for months if 
> not years. They should not, but they do. Getting at least the last pending 
> security updates out before turning the pipes off for good should be a 
> priority. It means they'd at least not be affected by those security issues, 
> only by the later ones.

The problem is not specific to EOL releases.

Security updates for other dist releases are affected in the same way. ;-)
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: yum: Critical path update in testing for 4 months?

[Michael Schwendt]

On Mon, 7 Dec 2015 10:53:32 -0700, Kevin Fenzi wrote:

> So, what you would like would be: 
> 
> a) If you have autokarma set and the update doesn't reach autokarma,
> push it automatically when it reaches the timeout. 

With a customisable timeout, it would be a good feature.

> b) Have another bodhi setting for 'push automatically when reaching the
> 'maintainer can now push to stable' time limit. (defaulted on or off?)

Not so good, because the current blocking period seems arbitrary.
14 days? 7 days? Why? It's a rather short period (considering that it
can take days for a mirror to pick up new packages) and doesn't give all
testers enough time to notice the test update and test it painstakingly.

> > What's needed is software developers and policy makers to agree that
> > some areas are problematic, and to agree on ideas and an agenda. To
> > agree that the karma system is flawed and things like testers
> > ignoring past votes and overriding another's -1 with their own +1
> > should not be possible.  
> 
> I don't think I agree with that as a blanket statement. What if the
> person who added -1 is just wrong, or you cannot duplicate the exact
> problem that they said the update had?

I wrote "testers _ignoring_ past votes", not testers disagreeing with
each other explicitly in their votes _and_ comments.

Who decides whether a person is wrong? And why can't such a decision not
require somebody to explicitly mask/override a wrong -1?

As it is so far, there are fights between people voting +1 and -1 like it
would be some sort of package popularity contest, and not even in the
comments the people refer to eachother being "wrong". Then, when adding a
comment and pointing at the QA feedback guidelines, people admit that they
haven't been aware of those guidelines.

> > How many layers of extra work do you ask for? Imagine that a fix is
> > from upstream or has been applied and released upstream already. What
> > extra testing and baby-sitting is needed at Fedora's side even for
> > entirely new packages?  
> 
> What would you propose?

A release process that makes sure that published updates (in particular
security fixes) reach the stable updates repo in adequate time.

> Ship the update directly to stable because upstream oked it?

Shipping directly into stable could be the right thing in _some_ cases.
Such as packages that never receive any tester feedback in bodhi.
I don't see why you would want upstream to ack any updates. If upstream
is not explicitly interested in Fedora's packages, there is no interest
in testing updates either.
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: yum: Critical path update in testing for 4 months?

[Adam Williamson]

On Mon, 2015-12-07 at 20:19 +0100, Michael Schwendt wrote:
> On Mon, 7 Dec 2015 10:53:32 -0700, Kevin Fenzi wrote:
> 
> > So, what you would like would be: 
> > 
> > a) If you have autokarma set and the update doesn't reach autokarma,
> > push it automatically when it reaches the timeout. 
> 
> With a customisable timeout, it would be a good feature.
> 
> > b) Have another bodhi setting for 'push automatically when reaching the
> > 'maintainer can now push to stable' time limit. (defaulted on or off?)
> 
> Not so good, because the current blocking period seems arbitrary.
> 14 days? 7 days? Why? It's a rather short period (considering that it
> can take days for a mirror to pick up new packages) and doesn't give all
> testers enough time to notice the test update and test it painstakingly.

It's a trade-off between people who want to be able to push updates out
promptly and people who want to ensure sufficient testing. It's never
possible to get a 'perfect' policy when dealing with the range of
packages in Fedora; no setup is ideal for *all* of them. All our
policies are trade-offs.

It shouldn't ever take 'days' for a mirror to pick up new packages. Any
mirrors that slow should be dropped from the mirror list, I think. The
timer starts from when the update is *pushed* to testing (i.e. when it
actually gets there on the master mirror), not from when the maintainer
*submits* it, so the delay that can occur between an update being
submitted and being pushed is not counted in the required wait time.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: yum: Critical path update in testing for 4 months?

[Kevin Fenzi]

On Sun, 6 Dec 2015 22:29:33 +0100
Michael Schwendt  wrote:

> On Sun, 6 Dec 2015 12:40:45 -0700, Kevin Fenzi wrote:
> 
> > Perhaps. But you are speculating that this is the case here. 
> > Unless you have talked to the maintainer and thats what they told
> > you?  
> 
> I'm not speculating as I didn't claim I know the reason why this
> particular Yum update is stuck.

Your email read that way to me at least. Sorry if I misunderstood you. 

> I answered to the general question "what is the reason for maintainers
> building updates without the intention to push them?". Knowing that
> some maintainers believe the release bureaucracy sucks can be helpful
> in understanding [some of] the background.

I suppose, but I think there's also other reasons. 
 
> I still talk to other packages from time to time, and I have learned
> to be careful when doing that, because some find it easier to
> complain about various things in a private conversation (and leave
> the project silently) instead of voicing their opinion in one of
> Fedora's public places.

I'm sorry to hear that. I personally don't mind feedback anywhere in
public, but that doesn't mean that I will agree with everything or
agree to do work to change things. 

...snip...

> It's only a lack of testers and a lack of _another method_ to publish
> such updates _automatically_ based on submitter's early request.

So, what you would like would be: 

a) If you have autokarma set and the update doesn't reach autokarma,
push it automatically when it reaches the timeout. 

or

b) Have another bodhi setting for 'push automatically when reaching the
'maintainer can now push to stable' time limit. (defaulted on or off?)

> What's needed is software developers and policy makers to agree that
> some areas are problematic, and to agree on ideas and an agenda. To
> agree that the karma system is flawed and things like testers
> ignoring past votes and overriding another's -1 with their own +1
> should not be possible.

I don't think I agree with that as a blanket statement. What if the
person who added -1 is just wrong, or you cannot duplicate the exact
problem that they said the update had?
 
> If people in Fedora leadership positions don't consider broken upgrade
> paths a problem, and the developers of update release tools don't
> consider them problematic either, not much will happen about such
> issues, for example. Users will continue to run into downgrades,
> unresolvable deps, or runtime breakage. And then there are all those
> ideas where the only response will be that patches will be accepted,
> with the ideas never making it onto a todo list/agenda.

upgrade path is a different issue. Perhaps it should be talked about in
a different thread instead of adding into this one thats not related to
it?

IMHO, upgrade path is less of an issue than it was in the past now that
upgrade methods use distro-sync. Of course there are cases where it
could be an issue still. 

> > > Currently I have two security fixes, which are two months old.
> > > Nobody does the needed testing. The karma isn't reached. Nobody
> > > ensures that they enter the stable updates repo even with 0
> > > karma. 
> > 
> > Perhaps you could solicit testers? Either upstream people or on the
> > test list or on irc?  
> 
> Perhaps Fedora is just not popular enough?

I don't see how you get to that conclusion from that problem statement.

> How many layers of extra work do you ask for? Imagine that a fix is
> from upstream or has been applied and released upstream already. What
> extra testing and baby-sitting is needed at Fedora's side even for
> entirely new packages?

What would you propose? Ship the update directly to stable because
upstream oked it?

> Remember the period when packagers voted +1 on their own updates.
> There still is no way to do that *officially*.  It is still assumed
> that packagers test their own update (even if they don't do that in
> Rawhide, uh-oh!), but they cannot ask for it to be pushed
> automatically after X days other than based on that karma threshold
> that won't be reached for some packages ever.

They could perhaps file a bodhi RFE asking for that feature?

> > > Meanwhile, F21
> > > has reached end-of-life without anyone making sure to do a last
> > > push of security fixes for it. 
> > 
> > We did do a last push. Just blindly pushing all security updates (if
> > they were ready or not) isn't a particularly good idea IMHO.   
> 
> Has the security team done anything at all to even push any other
> security updates for F21 not blindly? I mean, Fedora packagers
> receive all those security related tracker ticket bugzilla spam where
> the security team changes ticket fields again and again, but nobody
> cares that the released fixes find their way onto the Fedora User's
> installations?

You would be better addressing such questions to them: 

https://fedoraproject.org/wiki/Security_Team

I know they prioritized things and have been 

Re: yum: Critical path update in testing for 4 months?

[Reindl Harald]

Am 06.12.2015 um 15:30 schrieb Richard Shaw:

Usually I don't dig too deep but I saw a "critical path" update for yum
on F22 (which should be using dnf anyway).

So why is a critical path update with +10 karma still in testing?

(I know how, there's no auto push to stable based on karma for this update)


that's the reason that it did not happen automatically

but what is the reason for maintainers building updates without the 
intention to push them?


bodhi should punish maintaines with daily mails when a package has 
enough karma or has reached the time to get pushed even without karma so 
that the maintainer has a good reason for push it or decide to delete it 
(with a very good reason)




signature.asc
Description: OpenPGP digital signature
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: yum: Critical path update in testing for 4 months?

[Reindl Harald]

Am 06.12.2015 um 19:52 schrieb Michael Schwendt:

Currently I have two security fixes, which are two months old. Nobody
does the needed testing. The karma isn't reached. Nobody ensures that
they enter the stable updates repo even with 0 karma. Meanwhile, F21
has reached end-of-life without anyone making sure to do a last push
of security fixes for it. If I had not released any fixes, nobody would
have reminded me. In other cases, there have been CVE tickets in bugzilla
filed by the security team from Red Hat with nobody working on fixes for
Fedora, not even sending reminders. We need more thinking humans to make
the right decisions. Look at the age of updates in the updates-testing
reports! This is crap 2.0


that may all be true - BUT i have zero understanding for cases where i 
make a distr-upgrade, start as usually "fedora-easy-karma" and find a 
ton of "this package could be pushed to stable if the maintainer wishes" 
candidates


and for "There are maintainers, who dislike a lot of things related to 
the release processes. They consider bodhi a pain to use. They would 
prefer doing things differently, with less work, and more like 
fire'n'forget as how they do it within Rawhide" than frankly what's the 
point of prepare a update when one don't care about it? it's not a 
matter of like or disklike - it's a point of responsibility




signature.asc
Description: OpenPGP digital signature
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: yum: Critical path update in testing for 4 months?

[Reindl Harald]

Am 06.12.2015 um 21:20 schrieb Michael Schwendt:

And as you are not a package maintainer at Fedora, it could be that you
underestimate the amount of burden/bureaucracy that's considered
unnecessary by many packagers


explain me the burden/bureaucracy in the countless cases where 
"fedora-easy-karma" shows the status "this update has reached... and can 
be pushed to stable when the maintainer wishes" other then just press 
the button




signature.asc
Description: OpenPGP digital signature
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: yum: Critical path update in testing for 4 months?

[Michael Schwendt]

On Sun, 6 Dec 2015 21:47:24 +0100, Reindl Harald wrote:

> > And as you are not a package maintainer at Fedora, it could be that you
> > underestimate the amount of burden/bureaucracy that's considered
> > unnecessary by many packagers  
> 
> explain me the burden/bureaucracy in the countless cases where 
> "fedora-easy-karma" shows the status "this update has reached... and can 
> be pushed to stable when the maintainer wishes" other then just press 
> the button

Let me try.

Not specific to the Yum update you refer to in the original post, and I
don't know for how many updates the "autokarma" and critpath settings have
been lost upon migration to bodhi 2. => There are some strange status messages
in some of the tickets, such as very late +1 critpath votes.

Packager spends time on preparing an update for multiple dist targets,
test them locally, builds them within koji and waits for the builds to
become available in bodhi.

Packager submits the update and related bugzilla ticket numbers in bodhi
and either keeps the default karma threshold or adjusts it.

For non-critpath updates, some packagers enter a threshold of +1/-1 to
reduce testing requirements, but a single pet peeve reporter, who votes -1
due to a problem that exists for a much longer time already, can cause the
update to be withdrawn automatically. So, some packagers go with +1/-3 for
example, requiring a +1 from a single tester while still making it
possible to pull the update, if breakage is found by three testers.

After adding the updates to bodhi, packager wants to be done and wants to
move on to other tasks, since the update would be pushed automatically
*if* testers gave the required number of +1. Some of the votes could come
from the bug reporters, who are notified about the update inside their
bugzilla tickets. [I've pointed out that they are notified too early.]

[Packager knows that if an update for an older dist is published earlier
than an update for the current dist, this can lead to breaking dist
upgrades. Disabling karma based auto-pushing currently is the only way to
avoid that. More work, and reoccuring tasks to do manually aren't good.]

However, since the packager wanted to rely on karma based auto-pushing,
the email notification "This update has reached X days in testing and can
be pushed to stable now if the maintainer wishes" may come too early,
especially if the karma level has not been reached. The packager would
need to load the bodhi page linked in each of the notifications and decide
again whether to push manually without waiting for testers. Does the
packager want to push manually at that point? Sometimes. Not always.

You expect packagers to review all their bodhi tickets everytime they
receive a nagmail from bodhi, and then to decide again whether to wait for
testers or whether to push manually without any prior feedback. That doesn't
scale for everyone. The list of active updates inside the web ui can
grow easily, too.

Let's say you start reviewing your updates for F23. You decide to push an
update after 14 days and 0 karma, because you want to get it out as
quickly as policies allow. You must not forget the corresponding update
for F22. It has received a +1 and a -1 but has not reached -3 yet to
unpush it automatically. It may have been a better idea to unpush the
updates for all dists manually upon receiving the -1. But then what's the
point of a -3 threshold? And it also requires you to pay attention to
every bodhi mail you receive and take action immediately, or else the
messages about later votes and status changes may pile up quickly.

While waiting for updates to be pushed, the packager also needs to
be careful not to submit further updates. Bodhi used to be able to
obsolete older test updates automatically, which may be the wrong
thing to do in all cases (the new update may not be as good as the
current one and may require increased testing).

What would some packagers prefer?

A way to append updates into a queue and _be done_ unless a tester
intervenes, and if all goes well, the update gets published either after
positive feedback from a minimum number of testers _or_ after a grace
period of N days sitting in updates-testing.
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: yum: Critical path update in testing for 4 months?

[Adam Williamson]

On Sun, 2015-12-06 at 19:52 +0100, Michael Schwendt wrote:
> On Sun, 6 Dec 2015 15:50:10 +0100, Reindl Harald wrote:
> 
> > but what is the reason for maintainers building updates without the 
> > intention to push them?
> 
> There are maintainers, who dislike a lot of things related to the release
> processes. They consider bodhi a pain to use. They would prefer doing
> things differently, with less work, and more like fire'n'forget as how
> they do it within Rawhide.

That doesn't really add up. Auto-karma is the *default* in Bodhi. The
maintainer had to take specific action to disable it. If they want
things to be fire-and-forget, why disable auto-karma?

> Currently I have two security fixes, which are two months old. Nobody
> does the needed testing. The karma isn't reached.

If they're two months old, they can certainly be pushed with 0 karma.
The longest *any* update for *any* distro has to wait before it can be
pushed without karma is 2 weeks.

>  Nobody ensures that
> they enter the stable updates repo even with 0 karma. Meanwhile, F21
> has reached end-of-life without anyone making sure to do a last push
> of security fixes for it.

I've never understood why the idea of a 'last push of security fixes'
for an EOL release makes any sense. It's *EOL*. It doesn't matter if
there's a last-day push or not: everyone should stop using it the next
day, end of story. That's what EOL means.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: yum: Critical path update in testing for 4 months?

[Michael Schwendt]

On Sun, 06 Dec 2015 16:02:42 -0800, Adam Williamson wrote:

> On Sun, 2015-12-06 at 19:52 +0100, Michael Schwendt wrote:
> > On Sun, 6 Dec 2015 15:50:10 +0100, Reindl Harald wrote:
> >   
> > > but what is the reason for maintainers building updates without the 
> > > intention to push them?  
> > 
> > There are maintainers, who dislike a lot of things related to the release
> > processes. They consider bodhi a pain to use. They would prefer doing
> > things differently, with less work, and more like fire'n'forget as how
> > they do it within Rawhide.  
> 
> That doesn't really add up.

What doesn't add up?

> Auto-karma is the *default* in Bodhi. The
> maintainer had to take specific action to disable it. If they want
> things to be fire-and-forget, why disable auto-karma?

Nobody said that they do that. I refer to things bodhi cannot do today.
Keeping default karma settings is the closest thing to fire'n'forget,
except if karma threshold will not be reached.

> >  Nobody ensures that
> > they enter the stable updates repo even with 0 karma. Meanwhile, F21
> > has reached end-of-life without anyone making sure to do a last push
> > of security fixes for it.  
> 
> I've never understood why the idea of a 'last push of security fixes'
> for an EOL release makes any sense. It's *EOL*. It doesn't matter if
> there's a last-day push or not: everyone should stop using it the next
> day, end of story. That's what EOL means.

It need not happen on the "last day". It could have happened a month
before release.
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: yum: Critical path update in testing for 4 months?

[Michael Schwendt]

On Mon, 7 Dec 2015 00:08:37 +0100, Reindl Harald wrote:

> yes i expect that someone is able just read his mails and push a button 
> on a list in a webinterface

The productivity loss that's caused by people wading through email folders
every day is subject to studies.

Plus, by the time the first nagmail from bodhi is opened and read (after
filtering), the update may have been pushed already. Or the nagmail is
sent too early.

Publishing ready-to-ship updates is a reoccuring task that asks for more
automation.
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: yum: Critical path update in testing for 4 months?

[Kevin Kofler]

Adam Williamson wrote:
> I've never understood why the idea of a 'last push of security fixes'
> for an EOL release makes any sense. It's *EOL*. It doesn't matter if
> there's a last-day push or not: everyone should stop using it the next
> day, end of story. That's what EOL means.

Realistically, some people will still be running EOL releases for months if 
not years. They should not, but they do. Getting at least the last pending 
security updates out before turning the pipes off for good should be a 
priority. It means they'd at least not be affected by those security issues, 
only by the later ones.

Kevin Kofler
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: yum: Critical path update in testing for 4 months?

[Michael Schwendt]

On Sun, 6 Dec 2015 15:50:10 +0100, Reindl Harald wrote:

> but what is the reason for maintainers building updates without the 
> intention to push them?

There are maintainers, who dislike a lot of things related to the release
processes. They consider bodhi a pain to use. They would prefer doing
things differently, with less work, and more like fire'n'forget as how
they do it within Rawhide.

That's also the reason why they release another update while the previous
update has not been pushed yet. And bodhi still cannot handle that case
and sometimes pushes an older package after a newer package. If what I've
read somewhere is true, it isn't easy to fix in bodhi (or koji) for
reasons I don't know.

> bodhi should punish maintaines with daily mails when a package has 
> enough karma or has reached the time to get pushed even without karma so 
> that the maintainer has a good reason for push it or decide to delete it 
> (with a very good reason)

History has shown that attempts at "punishing" maintainers with so-called
"nagmails" doesn't lead to anything good. Automated systems have a bad habit
of sending nagmails when the human knows better. Then the human decides to
filter out the annoying mails.

Fedora's release process is poor and misdesigned and full of problems.

Currently I have two security fixes, which are two months old. Nobody
does the needed testing. The karma isn't reached. Nobody ensures that
they enter the stable updates repo even with 0 karma. Meanwhile, F21
has reached end-of-life without anyone making sure to do a last push
of security fixes for it. If I had not released any fixes, nobody would
have reminded me. In other cases, there have been CVE tickets in bugzilla
filed by the security team from Red Hat with nobody working on fixes for
Fedora, not even sending reminders. We need more thinking humans to make
the right decisions. Look at the age of updates in the updates-testing
reports! This is crap 2.0.
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: yum: Critical path update in testing for 4 months?

[Michael Schwendt]

On Sun, 6 Dec 2015 12:40:45 -0700, Kevin Fenzi wrote:

> Perhaps. But you are speculating that this is the case here. 
> Unless you have talked to the maintainer and thats what they told you?

I'm not speculating as I didn't claim I know the reason why this
particular Yum update is stuck.

I answered to the general question "what is the reason for maintainers
building updates without the intention to push them?". Knowing that some
maintainers believe the release bureaucracy sucks can be helpful in
understanding [some of] the background.

I still talk to other packages from time to time, and I have learned to be
careful when doing that, because some find it easier to complain about
various things in a private conversation (and leave the project silently)
instead of voicing their opinion in one of Fedora's public places.

> I think it far more likely that this is due to:
> https://github.com/fedora-infra/bodhi/issues/372
> which was/is a bug around the migration from bodhi1 to bodhi2 where
> some updates lost their setting of auto karma. 
> 
> If not that, then perhaps the maintainer wanted to be careful with this
> update for some reason and so wanted to push it manually. 
> 
> The only way to know for sure would be to ask.

Four months is a long time even for an update that is at karma 10 already.
Even if it's Yum, which has been broken before by updates that have been
rushed out too soon. If it fixes any bugs, waiting four months for the
fixes to get out is too long. [And a minor update to some fonts package
is pushed faster and more frequently. ;-)]

More strange, if it's an update that's at karma 0 after four months.
Bodhi forgetting about autokarma cannot be an issue in that case.

It's only a lack of testers and a lack of _another method_ to publish
such updates _automatically_ based on submitter's early request.

> > Fedora's release process is poor and misdesigned and full of problems.  
> 
> I'm sorry to hear you say so.
> 
> Do you have any ideas to improve things? Or would you prefer to
> continue to be a ray of sunshine when others ask for ideas?

What's needed is software developers and policy makers to agree that some
areas are problematic, and to agree on ideas and an agenda. To agree that
the karma system is flawed and things like testers ignoring past votes and
overriding another's -1 with their own +1 should not be possible.

If people in Fedora leadership positions don't consider broken upgrade
paths a problem, and the developers of update release tools don't consider
them problematic either, not much will happen about such issues, for example.
Users will continue to run into downgrades, unresolvable deps, or runtime
breakage. And then there are all those ideas where the only response will
be that patches will be accepted, with the ideas never making it onto a
todo list/agenda.

> > Currently I have two security fixes, which are two months old. Nobody
> > does the needed testing. The karma isn't reached. Nobody ensures that
> > they enter the stable updates repo even with 0 karma.   
> 
> Perhaps you could solicit testers? Either upstream people or on the
> test list or on irc?

Perhaps Fedora is just not popular enough?

How many layers of extra work do you ask for? Imagine that a fix is from
upstream or has been applied and released upstream already. What extra
testing and baby-sitting is needed at Fedora's side even for entirely new
packages?
Remember the period when packagers voted +1 on their own updates. There
still is no way to do that *officially*.  It is still assumed that
packagers test their own update (even if they don't do that in Rawhide,
uh-oh!), but they cannot ask for it to be pushed automatically after X days
other than based on that karma threshold that won't be reached for some
packages ever.

> > Meanwhile, F21
> > has reached end-of-life without anyone making sure to do a last push
> > of security fixes for it.   
> 
> We did do a last push. Just blindly pushing all security updates (if
> they were ready or not) isn't a particularly good idea IMHO. 

Has the security team done anything at all to even push any other security
updates for F21 not blindly? I mean, Fedora packagers receive all those
security related tracker ticket bugzilla spam where the security team
changes ticket fields again and again, but nobody cares that the released
fixes find their way onto the Fedora User's installations?
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: yum: Critical path update in testing for 4 months?

[Michael Schwendt]

On Sun, 6 Dec 2015 20:15:20 +0100, Reindl Harald wrote:

> BUT i have zero understanding for cases where i 
> make a distr-upgrade, start as usually "fedora-easy-karma" and find a 
> ton of "this package could be pushed to stable if the maintainer wishes" 
> candidates

You need to talk to the individual maintainers to find out.

Some have seen the nagmail they got from bodhi, but as they have set a low
karma threshold of 1 or 2 for the update, they wait for a +1 from testers.
And then the update would be published _automatically_. Others disable
those nagmails to begin with, since they only increase the "spam" in all
the cases where you really want to see feedback from somebody before
pushing an update.

It doesn't work well, if bug reporters wait for updates to appear in
the stable updates repo only to discover that the fix doesn't work or
that something else is broken. The bug reporter will be the first to
complain.

> and for "There are maintainers, who dislike a lot of things related to 
> the release processes. They consider bodhi a pain to use. They would 
> prefer doing things differently, with less work, and more like 
> fire'n'forget as how they do it within Rawhide" than frankly what's the 
> point of prepare a update when one don't care about it? it's not a 
> matter of like or disklike - it's a point of responsibility

This is cheap talk, and you shift responsibility to be a one-sided
thing placed on the maintainers' shoulders only. Bug reporters, who
enter something in bugzilla, but cannot be talked to, are not responsible
either.

And as you are not a package maintainer at Fedora, it could be that you
underestimate the amount of burden/bureaucracy that's considered
unnecessary by many packagers. There are so many updates where the
maintainer publishes an update confidently, but it still depends on
external factors, such as QA and critpath policies, unannounced changes
in deps, and the maintainer also must remember each and every detail of
any Updates policy there may be -- if the only goal is to get out a fix
asap.

One big problem here is that it's necessary to baby-sit an update for
too long and monitor its way into the repos, or else it may never find
its way out - and in some corner-cases, updates have been lost actually. ;-)

Odd as it may be, when I searched for my updates in the bodhi web ui,
I also discovered updates that were four months old already. I don't
think this has ever happened to me with the old bodhi.
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: yum: Critical path update in testing for 4 months?

[Reindl Harald]

Am 06.12.2015 um 22:29 schrieb Michael Schwendt:

What's needed is software developers and policy makers to agree that some
areas are problematic, and to agree on ideas and an agenda. To agree that
the karma system is flawed and things like testers ignoring past votes and
overriding another's -1 with their own +1 should not be possible


really?

i have seen more than once a -1 from somebody because *his* bug was not 
fixed by the update but others where - in that case *it is not* a 
regression and there is no point for -1


if i would get 1$ for every update which got a +1 from me while it did 
not fixed for me annyoing bugs *just because* they where *no regression* 
in *that* build i would be rich!




signature.asc
Description: OpenPGP digital signature
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: yum: Critical path update in testing for 4 months?

[Michael Schwendt]

On Sun, 6 Dec 2015 22:37:57 +0100, Reindl Harald wrote:

> i have seen more than once a -1 from somebody because *his* bug was not 
> fixed by the update but others where - in that case *it is not* a 
> regression and there is no point for -1

The Fedora Updates System is a place where to let out one's frustration
related to broken software/packages.

I had hoped that it would be obvious that I refer to the case where
somebody had voted -1 for a problem introduced by the update, but later
testers didn't read the comments and voted +1 nevertheless up to the
point of triggering an automatic push to stable.

It's covered by the guidelines:
https://fedoraproject.org/wiki/QA:Update_feedback_guidelines

Every -1 by somebody ought to stop autokarma based pushes and require
somebody to review the update ticket and decide whether to mask the -1. If
a new kernel fails for one tester, it would be wrong to push it to stable
only because it works for three other testers.
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: yum: Critical path update in testing for 4 months?

[Reindl Harald]

Am 06.12.2015 um 23:50 schrieb Michael Schwendt:

You expect packagers to review all their bodhi tickets everytime they
receive a nagmail from bodhi, and then to decide again whether to wait for
testers or whether to push manually without any prior feedback. That doesn't
scale for everyone. The list of active updates inside the web ui can
grow easily, too


yes, i expect this when i am able to follow all fedora and 7 other 
mailing-lists including bodhi-mails and give karma and review 100-400 
mail samples for bayes-training every day besides my daily jobs as the 
only sysadmin for more than 30 Fedora based servers, a few test-servers, 
4 fedora-based routers and get my *really* dayjob as software developer 
done at the same time


yes i expect that someone is able just read his mails and push a button 
on a list in a webinterface




signature.asc
Description: OpenPGP digital signature
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: yum: Critical path update in testing for 4 months?

[Kevin Fenzi]

On Sun, 6 Dec 2015 19:52:05 +0100
Michael Schwendt  wrote:

> On Sun, 6 Dec 2015 15:50:10 +0100, Reindl Harald wrote:
> 
> > but what is the reason for maintainers building updates without the 
> > intention to push them?  
> 
> There are maintainers, who dislike a lot of things related to the
> release processes. They consider bodhi a pain to use. They would
> prefer doing things differently, with less work, and more like
> fire'n'forget as how they do it within Rawhide.

Perhaps. But you are speculating that this is the case here. 
Unless you have talked to the maintainer and thats what they told you?

I think it far more likely that this is due to:
https://github.com/fedora-infra/bodhi/issues/372
which was/is a bug around the migration from bodhi1 to bodhi2 where
some updates lost their setting of auto karma. 

If not that, then perhaps the maintainer wanted to be careful with this
update for some reason and so wanted to push it manually. 

The only way to know for sure would be to ask.

> That's also the reason why they release another update while the
> previous update has not been pushed yet. And bodhi still cannot
> handle that case and sometimes pushes an older package after a newer
> package. If what I've read somewhere is true, it isn't easy to fix in
> bodhi (or koji) for reasons I don't know.

There's a possible fix for this that landed in the last bodhi update. 
It has to do with bodhi passing a bunch of packages to koji to tag,
and koji does not promise the order they will be tagged in. So, bodhi
has to know which is "newer" and split the operation up. 
 
> > bodhi should punish maintaines with daily mails when a package has 
> > enough karma or has reached the time to get pushed even without
> > karma so that the maintainer has a good reason for push it or
> > decide to delete it (with a very good reason)  

I think daily nag mails are over the top and anoying. 

> History has shown that attempts at "punishing" maintainers with
> so-called "nagmails" doesn't lead to anything good. Automated systems
> have a bad habit of sending nagmails when the human knows better.
> Then the human decides to filter out the annoying mails.
> 
> Fedora's release process is poor and misdesigned and full of problems.

I'm sorry to hear you say so.

Do you have any ideas to improve things? Or would you prefer to
continue to be a ray of sunshine when others ask for ideas?
 
> Currently I have two security fixes, which are two months old. Nobody
> does the needed testing. The karma isn't reached. Nobody ensures that
> they enter the stable updates repo even with 0 karma. 

Perhaps you could solicit testers? Either upstream people or on the
test list or on irc?

> Meanwhile, F21
> has reached end-of-life without anyone making sure to do a last push
> of security fixes for it. 

We did do a last push. Just blindly pushing all security updates (if
they were ready or not) isn't a particularly good idea IMHO. 

> If I had not released any fixes, nobody
> would have reminded me. In other cases, there have been CVE tickets
> in bugzilla filed by the security team from Red Hat with nobody
> working on fixes for Fedora, not even sending reminders. We need more
> thinking humans to make the right decisions. Look at the age of
> updates in the updates-testing reports! This is crap 2.0.

Yeah, the Fedora security sig has actually been ramping up trying to
deal with these. Perhaps you could offer your assistance there?

kevin




pgpMAQaqjN5r2.pgp
Description: OpenPGP digital signature
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: yum: Critical path update in testing for 4 months?

[Adam Williamson]

On Mon, 2015-12-07 at 02:07 +0100, Michael Schwendt wrote:
> On Sun, 06 Dec 2015 16:02:42 -0800, Adam Williamson wrote:
> 
> > On Sun, 2015-12-06 at 19:52 +0100, Michael Schwendt wrote:
> > > On Sun, 6 Dec 2015 15:50:10 +0100, Reindl Harald wrote:
> > >   
> > > > but what is the reason for maintainers building updates without the 
> > > > intention to push them?  
> > > 
> > > There are maintainers, who dislike a lot of things related to the release
> > > processes. They consider bodhi a pain to use. They would prefer doing
> > > things differently, with less work, and more like fire'n'forget as how
> > > they do it within Rawhide.  
> > 
> > That doesn't really add up.
> 
> What doesn't add up?

We were talking about a yum update that had been sitting around for
four months, with +10 karma. Your theory about why this kind of thing
happens doesn't match the facts of that situation: the update submitter
actually had to take specific action to make it so the update wouldn't
get pushed. If they were just 'fire'n'forget', the update would already
have gone stable weeks ago.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org