On 20/05/2020 01.06, John Snow wrote: > > > On 5/19/20 5:04 AM, Daniel P. Berrangé wrote: >> On Mon, May 18, 2020 at 03:56:36PM -0400, John Snow wrote: >>> >>> >>> On 5/15/20 6:23 AM, Daniel P. Berrangé wrote: >>>> On Fri, May 15, 2020 at 12:11:17PM +0200, Thomas Huth wrote: >>>>> On 07/04/2020 12.59, Philippe Mathieu-Daudé wrote: >>>>>> Hello, >>>>>> >>>>>> Following Markus thread on deprecating unmaintained (untested) code >>>>>> (machines) [1] and the effort done to gather the information shared in >>>>>> the replies [2], and the various acceptance tests added, is it >>>>>> feasible to require for the next release that each new device/machine >>>>>> is provided a test covering it? >>>>>> >>>>>> If no, what is missing? >>>>> >>>>> If a qtest is feasible, yes, I think we should require one for new >>>>> devices. But what about machines - you normally need a test image for >>>>> this. In that case, there is still the question where testing images >>>>> could be hosted. Not every developer has a web space where they could >>>>> put their test images onto. And what about images that contain non-free >>>>> code? >>>> >>>> Yep, it isn't feasible to make this a hard rule. >>>> >>>> IMHO this is where a support tier classification comes into play >>>> >>>> - Tier 1: actively maintained, qtest coverage available. Expected >>>> to work reliably at all times since every commit is CI >>>> tested >>>> >>>> - Tier 2: actively maintained, no qtest coverage. Should usually >>>> work but regression may creep in due to reliance on the >>>> maintainer to manually test on adhoc basis >>>> >>>> - Tier 3: not actively maintained, unknown state but liable to >>>> be broken indefinitely >>>> >>>> Tier 1 is obviously the most desirable state we would like everthing to >>>> be at. Contributors will have to fix problems their patches cause as >>>> they will be blocked by CI. >>>> >>>> Tier 2 is an admission that reality gets in the way. Ideally stuff in >>>> this tier will graduate to Tier 1 at some point. Even if it doesn't >>>> though, it is still valid to keep it in QEMU long term. Contributors >>>> shouldn't gratuitously break stuff in these board, but if they do, >>>> then the maintainer is ultimately responsible for fixing it, as the >>>> contributors don't have a test rig for it. >>>> >>>> Tier 3 is abandonware. If a maintainer doesn't appear, users should >>>> not expect it to continue to exist long term. Contributors are free >>>> to send patches which break this, and are under no obligation to >>>> fix problems in these boards. We may deprecate & delete it after a >>>> while >>>> >>>> >>>> Over time we'll likely add more criteria to stuff in Tier 1. This >>>> could lead to some things dropping from Tier 1 to Tier 2. This is >>>> OK, as it doesn't make those things worse than they already were. >>>> We're just saying that Tier 2 isn't as thoroughly tested as we >>>> would like it to be in an ideal world. >>> >>> I really like the idea of device support tiers codified directly in the >>> QEMU codebase, to give upstream users some idea of which devices we >>> expect to work and which we ... don't, really. >>> >>> Not every last device we offer is enterprise production ready, but we >>> don't necessarily do a good job of explaining which devices fall into >>> which categories, and we've got quite a few of them. >>> >>> I wonder if a 2.5th tier would be useful; something like a "hobbyist" >>> tier for pet project SoC boards and the like -- they're not abandoned, >>> but we also don't expect them to work, exactly. >>> >>> Mild semantic difference from Tier 3. >> >> I guess I was thinking such hobbyist stuff would fall into tier 2 if the >> hobbyist maintainer actually responds to fixing stuff, or tier 3 if they >> largely aren't active on the mailing list responding to issues/questions. >> >> We add have a 4 tier system overall and put hobbyist stuff at tier 3, >> and abandonware at tier 4. >> >> Probably shouldn't go beyond 4 tiers though, as the more criteria we add >> the harder it is to clearly decide which tier something should go into. >> >> The tier 1 vs 2 divison is clearly split based on CI which is a simple >> classification to decide on. >> >> The tier 2 vs 3 division is moderately clearly split based on whether >> there is a frequently active maintainer. >> >> We can probably squeeze in the 4th tier without too much ambiguity in >> the classisfication if we think it is adding something worthwhile either >> from our POV as maintainers, or for users consuming it. > > Yes, I didn't mean to start watering it down into a 1,380 tier system > that we're never able to properly utilize. > > I was thinking more along the lines of: > > - Device works and is well loved > - Device works and is well loved (but we have to test manually) > - Device doesn't work, but is well loved > (Use at your own peril, please file a bug report) > - Device doesn't work, and is unloved > > Perhaps it'd be clearer to name these Tier 1A, 1B, 2, and 3; where > things can shift from 1A to 1B as their test coverage allows, but it's > not meant to indicate general status otherwise.
All that sounds somewhat similar to the classification that we already use in our MAINTAINERS file - Supported, Maintained, Odd-Fixes, Orphan, Obsolete ... maybe we can avoid to introduce yet another classification system and merge the two (e.g. by also changing the classification system in MAINTAINERS a little bit?). Thomas