Re: [ovirt-devel] gerrit+ci improvement proposal
Il 03/06/2015 21:46, Eyal Edri ha scritto: - Original Message - From: Max Kovgan mkov...@redhat.com To: de...@ovirt.org Cc: infra@ovirt.org Sent: Wednesday, June 3, 2015 8:22:54 PM Subject: [ovirt-devel] gerrit+ci improvement proposal Hi everyone! We really want to have reliable and snappy CI: to allow short cycles and encourage developers to write tests. # Problem Many patches are neither ready for review nor for CI upon submission, which is OK. But running all the jobs on those patches with limited resources results in: overloaded resources, slow response time, unhappy developers. # Proposed Solution To run less jobs we know we don’t need to, thus making more resources for the jobs we need to run. We have been experimenting to make our CI stabler and quicker to respond by using gerrit flags. This has improved in both directions very well internally. Now it seems a good time to let all the oVirt projects to use this. This solution indirectly promotes reviews and quick tests - “to fail early”, yet full blown static code analysis and long tests to run “when ready”. # How it works 2 new gerrit independent flags are added to gerrit. ## CI flag Will express patch CI status. Values: * +1 CI passed * 0 CI did not run yet * -1 CI failed Permissions for setting: project maintainers (for special cases) should be able to set/override (except Jenkins). ## Workflow flag Will express patch “workflow” state. Values: * 0 Work In Progress * +1 Ready For Review * +2 Ready For Merge Permissions for setting: Owner can set +1, Project Maintainers can set +2 ## Review + CI Integration: Merging [“Submit” button to appear] will require: Review+1, CI+1, Workflow+2 Patch lifecycle now is: --- patch state |owner |reviewer |maintainer |CI tests |pass --- added/updated |- |-|- |quick|CI+1 review|Workflow+1|Review+1 |- |heavy |CI+1 merge ready |- |-|Workflow+2 |gating |CI+1 merge |- |-|merge |merge|CI+1 Changes from current workflow: Owner only adds reviewers, now owner needs to set Workflow+1 for the patch to be reviewed, and heavily auto-tested. Maintainer now needs to set Workflow+2 and wait for Submit button to appear after CI has completed running gating tests. Next step will be to automate merge the change after Workflow+2 has been set by the Maintainer and gating tests passed. ## Why now? It is elimination of waste. The sooner - the better. The solution has been used for a while and it works. Resolving the problem without gerrit involved will lead to adding unreliable code into jobs, and will still be prone to problems: Just recently, 3d ago we’ve tried detecting what to run from jenkins relying only on gerrit comments so that upon Verified+1, we’d run the job. We could not use “Review+1”, because it makes no sense at all, so we left the job to set Verified+1. Meaning - re-trigger itself immediately more than 1 times. Jenkins and its visitors very unhappy, and we had to stop those jobs, clean up the queue, and spam developers. ## OK OK OK. Now what? Now we want your comments and opinions before pushing this further: Please participate in this thread, so we can start trying it out. Ask, Suggest better ideas, all this is welcome. Best Regards! N.B. Of course, this is not written in stone, in case we find a better approach on solving those issues, we will change to it. And we will keep improving so don't be afraid that it will be enforced: if this does not work out we will discard it. P.S. Kudos to dcaro, most of the work was done by him, and most of this text too. +1 from me, releasing CI from running non critical and un-essential jobs will not only reduce load from ci, and shorted response time for developers, it will allow us to add much more powerful tests such as functional system tests that actually add hosts and run VMs, improving our ability to find regression much more effectively. Another benefit to consider is saving reviewers time. I.e not only jenkins benefits from Worklow+1, but also human reviewers. Instead of looking at a patch that is too early to be reviewed, the author can set the Workflow+1 when the code is ready to review (even if he didn't verified it yet), thus saving time to other reviewers - for example people can add an email rule to alert them only when they are added to patches that have Workflow+1, and not before. For human reviewers I suggest to keep using drafts until the patch is finished. Once it's finished and humans reviewed the logic of the patch, Workflow+1 should be triggered allowing automation to check the correctness of the patch. IMHO there's no reason for wasting CI time on patches that
Re: [ovirt-devel] gerrit+ci improvement proposal
- Original Message - From: Eyal Edri ee...@redhat.com To: Sandro Bonazzola sbona...@redhat.com Cc: infra@ovirt.org, de...@ovirt.org Sent: Thursday, June 4, 2015 9:46:40 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal - Original Message - From: Sandro Bonazzola sbona...@redhat.com To: Eyal Edri ee...@redhat.com, Max Kovgan mkov...@redhat.com Cc: de...@ovirt.org, infra@ovirt.org Sent: Thursday, June 4, 2015 9:11:10 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal Il 03/06/2015 21:46, Eyal Edri ha scritto: - Original Message - From: Max Kovgan mkov...@redhat.com To: de...@ovirt.org Cc: infra@ovirt.org Sent: Wednesday, June 3, 2015 8:22:54 PM Subject: [ovirt-devel] gerrit+ci improvement proposal Hi everyone! We really want to have reliable and snappy CI: to allow short cycles and encourage developers to write tests. # Problem Many patches are neither ready for review nor for CI upon submission, which is OK. But running all the jobs on those patches with limited resources results in: overloaded resources, slow response time, unhappy developers. # Proposed Solution To run less jobs we know we don’t need to, thus making more resources for the jobs we need to run. We have been experimenting to make our CI stabler and quicker to respond by using gerrit flags. This has improved in both directions very well internally. Now it seems a good time to let all the oVirt projects to use this. This solution indirectly promotes reviews and quick tests - “to fail early”, yet full blown static code analysis and long tests to run “when ready”. # How it works 2 new gerrit independent flags are added to gerrit. ## CI flag Will express patch CI status. Values: * +1 CI passed * 0 CI did not run yet * -1 CI failed Permissions for setting: project maintainers (for special cases) should be able to set/override (except Jenkins). ## Workflow flag Will express patch “workflow” state. Values: * 0 Work In Progress * +1 Ready For Review * +2 Ready For Merge Permissions for setting: Owner can set +1, Project Maintainers can set +2 ## Review + CI Integration: Merging [“Submit” button to appear] will require: Review+1, CI+1, Workflow+2 Patch lifecycle now is: --- patch state |owner |reviewer |maintainer |CI tests |pass --- added/updated |- |-|- |quick|CI+1 review|Workflow+1|Review+1 |- |heavy |CI+1 merge ready |- |-|Workflow+2 |gating |CI+1 merge |- |-|merge |merge|CI+1 Changes from current workflow: Owner only adds reviewers, now owner needs to set Workflow+1 for the patch to be reviewed, and heavily auto-tested. Maintainer now needs to set Workflow+2 and wait for Submit button to appear after CI has completed running gating tests. Next step will be to automate merge the change after Workflow+2 has been set by the Maintainer and gating tests passed. ## Why now? It is elimination of waste. The sooner - the better. The solution has been used for a while and it works. Resolving the problem without gerrit involved will lead to adding unreliable code into jobs, and will still be prone to problems: Just recently, 3d ago we’ve tried detecting what to run from jenkins relying only on gerrit comments so that upon Verified+1, we’d run the job. We could not use “Review+1”, because it makes no sense at all, so we left the job to set Verified+1. Meaning - re-trigger itself immediately more than 1 times. Jenkins and its visitors very unhappy, and we had to stop those jobs, clean up the queue, and spam developers. ## OK OK OK. Now what? Now we want your comments and opinions before pushing this further: Please participate in this thread, so we can start trying it out. Ask, Suggest better ideas, all this is welcome. Best Regards! N.B. Of course, this is not written in stone, in case we find a better approach on solving those issues, we will change to it. And we will keep improving so don't be afraid that it will be enforced: if this does not work out we will discard it. P.S. Kudos to dcaro, most of the work was done by him, and most of this text too. +1 from me, releasing CI from running non critical and un-essential jobs will not only reduce load from ci, and shorted response time for developers, it will allow us to add much more powerful tests such as functional system tests that actually add hosts and run VMs, improving
Logwatch for linode01.ovirt.org (Linux)
### Logwatch 7.3.6 (05/19/07) Processing Initiated: Thu Jun 4 03:13:47 2015 Date Range Processed: yesterday ( 2015-Jun-03 ) Period is day. Detail Level of Output: 0 Type of Output: unformatted Logfiles for Host: linode01.ovirt.org ## - httpd Begin Requests with error response codes 400 Bad Request /: 1 Time(s) /admin/index.php: 1 Time(s) /bitrix/admin/index.php?lang=en: 1 Time(s) 404 Not Found /.libs.php?0: 3 Time(s) //components/com_jnews/includes/openflashc ... =cpx.php.phpgif: 2 Time(s) //components/com_jnews/includes/openflashc ... hp?name=cpx.php: 2 Time(s) //components/com_jnews/includes/openflashc ... p?name=petx.php: 2 Time(s) //components/com_jnews/includes/openflashc ... petx.php.phpgif: 2 Time(s) /Diagnostics.asp: 75 Time(s) /Ringing.at.your.dorbell!: 75 Time(s) /mailman/listinfo/users: 1 Time(s) /__mailman/listinfo/engine-devel: 1 Time(s) /_h5ai/client/css/administrator/index.php: 1 Time(s) /_h5ai/client/css/wp-login.php: 1 Time(s) /_h5ai/client/images/app-16x16.ico: 24 Time(s) /_h5ai/client/images/favicon/administrator/index.php: 1 Time(s) /_h5ai/client/images/favicon/wp-login.php: 1 Time(s) /admin.php: 5 Time(s) /admin/: 5 Time(s) /admin/board: 2 Time(s) /admin/login.php: 5 Time(s) /administrator/components/com_acymailing/i ... ?name=magic.php: 9 Time(s) /administrator/components/com_acymailing/i ... e=magic.php.pHp: 9 Time(s) /administrator/components/com_civicrm/civi ... ?name=magic.php: 9 Time(s) /administrator/components/com_civicrm/civi ... e=magic.php.pHp: 9 Time(s) /administrator/components/com_jinc/classes ... ?name=magic.php: 8 Time(s) /administrator/components/com_jinc/classes ... e=magic.php.pHp: 8 Time(s) /administrator/components/com_jnews/includ ... ?name=magic.php: 7 Time(s) /administrator/components/com_jnews/includ ... e=magic.php.pHp: 8 Time(s) /administrator/components/com_jnewsletter/ ... ?name=magic.php: 9 Time(s) /administrator/components/com_jnewsletter/ ... e=magic.php.pHp: 9 Time(s) /administrator/components/com_maian15/char ... ?name=magic.php: 8 Time(s) /administrator/components/com_maian15/char ... e=magic.php.pHp: 8 Time(s) /administrator/index.php: 6 Time(s) /apple-touch-icon-precomposed.png: 3 Time(s) /apple-touch-icon.png: 3 Time(s) /bitrix/admin/index.php?lang=en: 5 Time(s) /blog/wp-admin/: 25 Time(s) /board: 4 Time(s) /category/news/feed: 1 Time(s) /category/news/feed/: 6 Time(s) /components/com_acymailing/inc/openflash/p ... ?name=magic.php: 9 Time(s) /components/com_acymailing/inc/openflash/p ... e=magic.php.pHp: 9 Time(s) /components/com_civicrm/civicrm/packages/O ... ?name=magic.php: 9 Time(s) /components/com_civicrm/civicrm/packages/O ... e=magic.php.pHp: 9 Time(s) /components/com_jinc/classes/graphics/php- ... ?name=magic.php: 7 Time(s) /components/com_jinc/classes/graphics/php- ... e=magic.php.pHp: 9 Time(s) /components/com_jnews/includes/openflashch ... ?name=magic.php: 7 Time(s) /components/com_jnews/includes/openflashch ... e=magic.php.pHp: 8 Time(s) /components/com_jnewsletter/includes/openf ... ?name=magic.php: 9 Time(s) /components/com_jnewsletter/includes/openf ... e=magic.php.pHp: 9 Time(s) /components/com_maian15/charts/php-ofc-lib ... ?name=magic.php: 8 Time(s) /components/com_maian15/charts/php-ofc-lib ... e=magic.php.pHp: 7 Time(s) /favicon.ico: 687 Time(s) /flashgallery.php?sohai: 3 Time(s) /html.php?0: 3 Time(s) /index.php/forums/member/register: 1 Time(s) /index.php?gf_page=upload: 2 Time(s) /listinfo/board: 2 Time(s) /m/pipermail/users/2014-January/020652.html: 2 Time(s) /mailman/listin: 1 Time(s) /mobile/pipermail/kimchi-devel/2015-January/009543.html: 1 Time(s) /mobile/pipermail/users/2012-August/009196.html: 1 Time(s) /mobile/pipermail/users/2013-January/011483.html: 1 Time(s) /old/wp-admin/: 24 Time(s) /pipermail/engine-patches/2011-December/001690.html: 1 Time(s) /pipermail/engine-patches/2011-November/000250.html: 1 Time(s) /pipermail/engine-patches/2012-October/038644.html: 1 Time(s) /pipermail/engine-patches/2013-June/079724.html: 1 Time(s) /pipermail/engine-patches/2013-November/subject.html: 5 Time(s) /pipermail/engine-patches/2014-April/175131.html: 1 Time(s) /pipermail/engine-patches/2014-March/subject.html: 4 Time(s) /pipermail/forum.php: 1 Time(s)
Re: [ovirt-devel] gerrit+ci improvement proposal
- Original Message - From: Nir Soffer nsof...@redhat.com To: Max Kovgan mkov...@redhat.com Cc: infra@ovirt.org, de...@ovirt.org Sent: Thursday, June 4, 2015 10:26:22 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal - Original Message - From: Max Kovgan mkov...@redhat.com To: de...@ovirt.org Cc: infra@ovirt.org Sent: Wednesday, June 3, 2015 8:22:54 PM Subject: [ovirt-devel] gerrit+ci improvement proposal Hi everyone! We really want to have reliable and snappy CI: to allow short cycles and encourage developers to write tests. # Problem Many patches are neither ready for review nor for CI upon submission, which is OK. But running all the jobs on those patches with limited resources results in: overloaded resources, slow response time, unhappy developers. # Proposed Solution To run less jobs we know we don’t need to, thus making more resources for the jobs we need to run. We have been experimenting to make our CI stabler and quicker to respond by using gerrit flags. This has improved in both directions very well internally. Now it seems a good time to let all the oVirt projects to use this. This solution indirectly promotes reviews and quick tests - “to fail early”, yet full blown static code analysis and long tests to run “when ready”. # How it works 2 new gerrit independent flags are added to gerrit. ## CI flag Will express patch CI status. Values: * +1 CI passed * 0 CI did not run yet * -1 CI failed -1 You must have CI error state. Most of the errors I have seen, the CI fail to run, and the failure is not related to the tested patch. its not possible to populate the exact error into the gerrit comment, that's why there is a link to the job. the fact that you've seen false positive errors is unfourtunate, but it doesn't represent the majority of patches, and each false positive error should be communicated to infra team so we can resolve it, it has nothing to do with this suggestion, on the contrary - once jenkins will run much less un-needed runs on patches, the probability for false positives will reduce dramatically. Permissions for setting: project maintainers (for special cases) should be able to set/override (except Jenkins). ## Workflow flag Will express patch “workflow” state. Values: * 0 Work In Progress * +1 Ready For Review * +2 Ready For Merge Permissions for setting: Owner can set +1, Project Maintainers can set +2 ## Review + CI Integration: Merging [“Submit” button to appear] will require: Review+1, CI+1, Workflow+2 Patch lifecycle now is: --- patch state |owner |reviewer |maintainer |CI tests |pass --- added/updated |- |-|- |quick|CI+1 review|Workflow+1|Review+1 |- |heavy|CI+1 merge ready |- |-|Workflow+2 |gating |CI+1 merge |- |-|merge |merge|CI+1 -1 Cannot require CI+1, the CI is not reliable enough yet. Even if the CI will be reliable, a failed test which is not related to the submitted patch should not block unrelated changes. you're repeating the same calim from above, same answer. Changes from current workflow: Owner only adds reviewers, now owner needs to set Workflow+1 for the patch to be reviewed, and heavily auto-tested. Maintainer now needs to set Workflow+2 and wait for Submit button to appear after CI has completed running gating tests. -1 This means more work for developers. Instead, make workflow default to +1. If the want to disable the CI, she can set it to 0. this will serve nothing, and jobs will keep running on each patchset as now, increasing the time you need to wait for each job to run and adding risk for failures. the only alternative to this is to start using Drafts for in progress patches, but only 1/4 of the patches are drafts atm, so we still are running more jobs on ci than we should, thus preventing us from running more complex and useful jobs. Developers need more power and less process overhead. Nir ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: [ovirt-devel] gerrit+ci improvement proposal
- Original Message - From: Sandro Bonazzola sbona...@redhat.com To: Eyal Edri ee...@redhat.com Cc: infra@ovirt.org, de...@ovirt.org Sent: Thursday, June 4, 2015 9:50:56 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal Il 04/06/2015 08:46, Eyal Edri ha scritto: - Original Message - From: Sandro Bonazzola sbona...@redhat.com To: Eyal Edri ee...@redhat.com, Max Kovgan mkov...@redhat.com Cc: de...@ovirt.org, infra@ovirt.org Sent: Thursday, June 4, 2015 9:11:10 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal Il 03/06/2015 21:46, Eyal Edri ha scritto: - Original Message - From: Max Kovgan mkov...@redhat.com To: de...@ovirt.org Cc: infra@ovirt.org Sent: Wednesday, June 3, 2015 8:22:54 PM Subject: [ovirt-devel] gerrit+ci improvement proposal Hi everyone! We really want to have reliable and snappy CI: to allow short cycles and encourage developers to write tests. # Problem Many patches are neither ready for review nor for CI upon submission, which is OK. But running all the jobs on those patches with limited resources results in: overloaded resources, slow response time, unhappy developers. # Proposed Solution To run less jobs we know we don’t need to, thus making more resources for the jobs we need to run. We have been experimenting to make our CI stabler and quicker to respond by using gerrit flags. This has improved in both directions very well internally. Now it seems a good time to let all the oVirt projects to use this. This solution indirectly promotes reviews and quick tests - “to fail early”, yet full blown static code analysis and long tests to run “when ready”. # How it works 2 new gerrit independent flags are added to gerrit. ## CI flag Will express patch CI status. Values: * +1 CI passed * 0 CI did not run yet * -1 CI failed Permissions for setting: project maintainers (for special cases) should be able to set/override (except Jenkins). ## Workflow flag Will express patch “workflow” state. Values: * 0 Work In Progress * +1 Ready For Review * +2 Ready For Merge Permissions for setting: Owner can set +1, Project Maintainers can set +2 ## Review + CI Integration: Merging [“Submit” button to appear] will require: Review+1, CI+1, Workflow+2 Patch lifecycle now is: --- patch state |owner |reviewer |maintainer |CI tests |pass --- added/updated |- |-|- |quick|CI+1 review|Workflow+1|Review+1 |- |heavy |CI+1 merge ready |- |-|Workflow+2 |gating |CI+1 merge |- |-|merge |merge|CI+1 Changes from current workflow: Owner only adds reviewers, now owner needs to set Workflow+1 for the patch to be reviewed, and heavily auto-tested. Maintainer now needs to set Workflow+2 and wait for Submit button to appear after CI has completed running gating tests. Next step will be to automate merge the change after Workflow+2 has been set by the Maintainer and gating tests passed. ## Why now? It is elimination of waste. The sooner - the better. The solution has been used for a while and it works. Resolving the problem without gerrit involved will lead to adding unreliable code into jobs, and will still be prone to problems: Just recently, 3d ago we’ve tried detecting what to run from jenkins relying only on gerrit comments so that upon Verified+1, we’d run the job. We could not use “Review+1”, because it makes no sense at all, so we left the job to set Verified+1. Meaning - re-trigger itself immediately more than 1 times. Jenkins and its visitors very unhappy, and we had to stop those jobs, clean up the queue, and spam developers. ## OK OK OK. Now what? Now we want your comments and opinions before pushing this further: Please participate in this thread, so we can start trying it out. Ask, Suggest better ideas, all this is welcome. Best Regards! N.B. Of course, this is not written in stone, in case we find a better approach on solving those issues, we will change to it. And we will keep improving so don't be afraid that it will be enforced: if this does not work out we will discard it. P.S. Kudos to dcaro, most of the work was done by him, and most of this text too. +1 from me, releasing CI from running non critical and un-essential jobs will not only reduce load from ci, and shorted response time for developers, it will allow us to add much more powerful tests such as functional system tests that actually add hosts and run VMs, improving our ability to find regression much more effectively. Another benefit to consider is
RE: Access to ovirt-srv08.ovirt.org
The question is the nesting depth.We run currently in depth 2 .VM inside VM.Will depth of 3 will work.VM slave, and another 2 VM nesting above it.KVM people? Plz help.Tolik. Original message From: Eyal Edri ee...@redhat.com Date: 03/06/2015 11:14 PM (GMT+02:00) To: tlito...@redhat.com Cc: infra@ovirt.org Subject: Re: Access to ovirt-srv08.ovirt.org I'd like to move the system testing framework to usenested vms instead of directly using the baremetal hosts.we did use a single host to POC the system, i think now that we'reabout to expand it will be much more managble and scalable if we'llstart using nested VMs which will be managed via the hosted engine.e.- Original Message - From: "Tolik Litvosky" tlito...@redhat.com To: infra@ovirt.org Sent: Wednesday, June 3, 2015 3:59:36 PM Subject: Access to ovirt-srv08.ovirt.org Hi I need access to the ovirt-srv08.ovirt.org in order to update the testenv CI repo directory. -- Best regards Tolik Litovsky RHEV-H Team Red Hat Red Hat: trustworthy, transformative technology. Powered by the community. Connect at redhat.com ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra smime.p7s Description: S/MIME Cryptographic Signature ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: [ovirt-devel] gerrit+ci improvement proposal
- Original Message - From: Sandro Bonazzola sbona...@redhat.com To: Eyal Edri ee...@redhat.com, Max Kovgan mkov...@redhat.com Cc: de...@ovirt.org, infra@ovirt.org Sent: Thursday, June 4, 2015 9:11:10 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal Il 03/06/2015 21:46, Eyal Edri ha scritto: - Original Message - From: Max Kovgan mkov...@redhat.com To: de...@ovirt.org Cc: infra@ovirt.org Sent: Wednesday, June 3, 2015 8:22:54 PM Subject: [ovirt-devel] gerrit+ci improvement proposal Hi everyone! We really want to have reliable and snappy CI: to allow short cycles and encourage developers to write tests. # Problem Many patches are neither ready for review nor for CI upon submission, which is OK. But running all the jobs on those patches with limited resources results in: overloaded resources, slow response time, unhappy developers. # Proposed Solution To run less jobs we know we don’t need to, thus making more resources for the jobs we need to run. We have been experimenting to make our CI stabler and quicker to respond by using gerrit flags. This has improved in both directions very well internally. Now it seems a good time to let all the oVirt projects to use this. This solution indirectly promotes reviews and quick tests - “to fail early”, yet full blown static code analysis and long tests to run “when ready”. # How it works 2 new gerrit independent flags are added to gerrit. ## CI flag Will express patch CI status. Values: * +1 CI passed * 0 CI did not run yet * -1 CI failed Permissions for setting: project maintainers (for special cases) should be able to set/override (except Jenkins). ## Workflow flag Will express patch “workflow” state. Values: * 0 Work In Progress * +1 Ready For Review * +2 Ready For Merge Permissions for setting: Owner can set +1, Project Maintainers can set +2 ## Review + CI Integration: Merging [“Submit” button to appear] will require: Review+1, CI+1, Workflow+2 Patch lifecycle now is: --- patch state |owner |reviewer |maintainer |CI tests |pass --- added/updated |- |-|- |quick|CI+1 review|Workflow+1|Review+1 |- |heavy |CI+1 merge ready |- |-|Workflow+2 |gating |CI+1 merge |- |-|merge |merge|CI+1 Changes from current workflow: Owner only adds reviewers, now owner needs to set Workflow+1 for the patch to be reviewed, and heavily auto-tested. Maintainer now needs to set Workflow+2 and wait for Submit button to appear after CI has completed running gating tests. Next step will be to automate merge the change after Workflow+2 has been set by the Maintainer and gating tests passed. ## Why now? It is elimination of waste. The sooner - the better. The solution has been used for a while and it works. Resolving the problem without gerrit involved will lead to adding unreliable code into jobs, and will still be prone to problems: Just recently, 3d ago we’ve tried detecting what to run from jenkins relying only on gerrit comments so that upon Verified+1, we’d run the job. We could not use “Review+1”, because it makes no sense at all, so we left the job to set Verified+1. Meaning - re-trigger itself immediately more than 1 times. Jenkins and its visitors very unhappy, and we had to stop those jobs, clean up the queue, and spam developers. ## OK OK OK. Now what? Now we want your comments and opinions before pushing this further: Please participate in this thread, so we can start trying it out. Ask, Suggest better ideas, all this is welcome. Best Regards! N.B. Of course, this is not written in stone, in case we find a better approach on solving those issues, we will change to it. And we will keep improving so don't be afraid that it will be enforced: if this does not work out we will discard it. P.S. Kudos to dcaro, most of the work was done by him, and most of this text too. +1 from me, releasing CI from running non critical and un-essential jobs will not only reduce load from ci, and shorted response time for developers, it will allow us to add much more powerful tests such as functional system tests that actually add hosts and run VMs, improving our ability to find regression much more effectively. Another benefit to consider is saving reviewers time. I.e not only jenkins benefits from Worklow+1, but also human reviewers. Instead of looking at a patch that is too early to be reviewed, the author can set the Workflow+1 when the code is ready to review (even if he didn't verified it yet), thus saving time to other reviewers -
Re: Access to ovirt-srv08.ovirt.org
- Original Message - From: Tolik Litovsky tlito...@redhat.com To: ee...@redhat.com Cc: infra@ovirt.org Sent: Thursday, June 4, 2015 9:28:27 AM Subject: RE: Access to ovirt-srv08.ovirt.org The question is the nesting depth. We run currently in depth 2 . VM inside VM. Will depth of 3 will work. VM slave, and another 2 VM nesting above it. KVM people? Plz help. adding devel, but please elaborate on the depth, does it have to be nested or you're doing it because you only had 1 bare metal host till now (think you'll have 2 nested vms for example) Tolik. Original message From: Eyal Edri ee...@redhat.com Date: 03/06/2015 11:14 PM (GMT+02:00) To: tlito...@redhat.com Cc: infra@ovirt.org Subject: Re: Access to ovirt-srv08.ovirt.org I'd like to move the system testing framework to use nested vms instead of directly using the baremetal hosts. we did use a single host to POC the system, i think now that we're about to expand it will be much more managble and scalable if we'll start using nested VMs which will be managed via the hosted engine. e. - Original Message - From: Tolik Litvosky tlito...@redhat.com To: infra@ovirt.org Sent: Wednesday, June 3, 2015 3:59:36 PM Subject: Access to ovirt-srv08.ovirt.org Hi I need access to the ovirt-srv08.ovirt.org in order to update the testenv CI repo directory. -- Best regards Tolik Litovsky RHEV-H Team Red Hat Red Hat: trustworthy, transformative technology. Powered by the community. Connect at redhat.com ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: [ovirt-devel] gerrit+ci improvement proposal
Il 04/06/2015 08:46, Eyal Edri ha scritto: - Original Message - From: Sandro Bonazzola sbona...@redhat.com To: Eyal Edri ee...@redhat.com, Max Kovgan mkov...@redhat.com Cc: de...@ovirt.org, infra@ovirt.org Sent: Thursday, June 4, 2015 9:11:10 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal Il 03/06/2015 21:46, Eyal Edri ha scritto: - Original Message - From: Max Kovgan mkov...@redhat.com To: de...@ovirt.org Cc: infra@ovirt.org Sent: Wednesday, June 3, 2015 8:22:54 PM Subject: [ovirt-devel] gerrit+ci improvement proposal Hi everyone! We really want to have reliable and snappy CI: to allow short cycles and encourage developers to write tests. # Problem Many patches are neither ready for review nor for CI upon submission, which is OK. But running all the jobs on those patches with limited resources results in: overloaded resources, slow response time, unhappy developers. # Proposed Solution To run less jobs we know we don’t need to, thus making more resources for the jobs we need to run. We have been experimenting to make our CI stabler and quicker to respond by using gerrit flags. This has improved in both directions very well internally. Now it seems a good time to let all the oVirt projects to use this. This solution indirectly promotes reviews and quick tests - “to fail early”, yet full blown static code analysis and long tests to run “when ready”. # How it works 2 new gerrit independent flags are added to gerrit. ## CI flag Will express patch CI status. Values: * +1 CI passed * 0 CI did not run yet * -1 CI failed Permissions for setting: project maintainers (for special cases) should be able to set/override (except Jenkins). ## Workflow flag Will express patch “workflow” state. Values: * 0 Work In Progress * +1 Ready For Review * +2 Ready For Merge Permissions for setting: Owner can set +1, Project Maintainers can set +2 ## Review + CI Integration: Merging [“Submit” button to appear] will require: Review+1, CI+1, Workflow+2 Patch lifecycle now is: --- patch state |owner |reviewer |maintainer |CI tests |pass --- added/updated |- |-|- |quick|CI+1 review|Workflow+1|Review+1 |- |heavy |CI+1 merge ready |- |-|Workflow+2 |gating |CI+1 merge |- |-|merge |merge|CI+1 Changes from current workflow: Owner only adds reviewers, now owner needs to set Workflow+1 for the patch to be reviewed, and heavily auto-tested. Maintainer now needs to set Workflow+2 and wait for Submit button to appear after CI has completed running gating tests. Next step will be to automate merge the change after Workflow+2 has been set by the Maintainer and gating tests passed. ## Why now? It is elimination of waste. The sooner - the better. The solution has been used for a while and it works. Resolving the problem without gerrit involved will lead to adding unreliable code into jobs, and will still be prone to problems: Just recently, 3d ago we’ve tried detecting what to run from jenkins relying only on gerrit comments so that upon Verified+1, we’d run the job. We could not use “Review+1”, because it makes no sense at all, so we left the job to set Verified+1. Meaning - re-trigger itself immediately more than 1 times. Jenkins and its visitors very unhappy, and we had to stop those jobs, clean up the queue, and spam developers. ## OK OK OK. Now what? Now we want your comments and opinions before pushing this further: Please participate in this thread, so we can start trying it out. Ask, Suggest better ideas, all this is welcome. Best Regards! N.B. Of course, this is not written in stone, in case we find a better approach on solving those issues, we will change to it. And we will keep improving so don't be afraid that it will be enforced: if this does not work out we will discard it. P.S. Kudos to dcaro, most of the work was done by him, and most of this text too. +1 from me, releasing CI from running non critical and un-essential jobs will not only reduce load from ci, and shorted response time for developers, it will allow us to add much more powerful tests such as functional system tests that actually add hosts and run VMs, improving our ability to find regression much more effectively. Another benefit to consider is saving reviewers time. I.e not only jenkins benefits from Worklow+1, but also human reviewers. Instead of looking at a patch that is too early to be reviewed, the author can set the Workflow+1 when the code is ready to review (even if he didn't verified it yet), thus saving time to other reviewers - for example people can add an email rule to alert them only when they are added to patches that
Re: [ovirt-devel] gerrit+ci improvement proposal
On 06/04, Eyal Edri wrote: - Original Message - From: Sandro Bonazzola sbona...@redhat.com To: Eyal Edri ee...@redhat.com Cc: infra@ovirt.org, de...@ovirt.org Sent: Thursday, June 4, 2015 9:50:56 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal Il 04/06/2015 08:46, Eyal Edri ha scritto: - Original Message - From: Sandro Bonazzola sbona...@redhat.com To: Eyal Edri ee...@redhat.com, Max Kovgan mkov...@redhat.com Cc: de...@ovirt.org, infra@ovirt.org Sent: Thursday, June 4, 2015 9:11:10 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal Il 03/06/2015 21:46, Eyal Edri ha scritto: - Original Message - From: Max Kovgan mkov...@redhat.com To: de...@ovirt.org Cc: infra@ovirt.org Sent: Wednesday, June 3, 2015 8:22:54 PM Subject: [ovirt-devel] gerrit+ci improvement proposal Hi everyone! We really want to have reliable and snappy CI: to allow short cycles and encourage developers to write tests. # Problem Many patches are neither ready for review nor for CI upon submission, which is OK. But running all the jobs on those patches with limited resources results in: overloaded resources, slow response time, unhappy developers. # Proposed Solution To run less jobs we know we don’t need to, thus making more resources for the jobs we need to run. We have been experimenting to make our CI stabler and quicker to respond by using gerrit flags. This has improved in both directions very well internally. Now it seems a good time to let all the oVirt projects to use this. This solution indirectly promotes reviews and quick tests - “to fail early”, yet full blown static code analysis and long tests to run “when ready”. # How it works 2 new gerrit independent flags are added to gerrit. ## CI flag Will express patch CI status. Values: * +1 CI passed * 0 CI did not run yet * -1 CI failed Permissions for setting: project maintainers (for special cases) should be able to set/override (except Jenkins). ## Workflow flag Will express patch “workflow” state. Values: * 0 Work In Progress * +1 Ready For Review * +2 Ready For Merge Permissions for setting: Owner can set +1, Project Maintainers can set +2 ## Review + CI Integration: Merging [“Submit” button to appear] will require: Review+1, CI+1, Workflow+2 Patch lifecycle now is: --- patch state |owner |reviewer |maintainer |CI tests |pass --- added/updated |- |-|- |quick|CI+1 review|Workflow+1|Review+1 |- |heavy |CI+1 merge ready |- |-|Workflow+2 |gating |CI+1 merge |- |-|merge |merge|CI+1 Changes from current workflow: Owner only adds reviewers, now owner needs to set Workflow+1 for the patch to be reviewed, and heavily auto-tested. Maintainer now needs to set Workflow+2 and wait for Submit button to appear after CI has completed running gating tests. Next step will be to automate merge the change after Workflow+2 has been set by the Maintainer and gating tests passed. ## Why now? It is elimination of waste. The sooner - the better. The solution has been used for a while and it works. Resolving the problem without gerrit involved will lead to adding unreliable code into jobs, and will still be prone to problems: Just recently, 3d ago we’ve tried detecting what to run from jenkins relying only on gerrit comments so that upon Verified+1, we’d run the job. We could not use “Review+1”, because it makes no sense at all, so we left the job to set Verified+1. Meaning - re-trigger itself immediately more than 1 times. Jenkins and its visitors very unhappy, and we had to stop those jobs, clean up the queue, and spam developers. ## OK OK OK. Now what? Now we want your comments and opinions before pushing this further: Please participate in this thread, so we can start trying it out. Ask, Suggest better ideas, all this is welcome. Best Regards! N.B. Of course, this is not written in stone, in case we find a better approach on solving those issues, we will change to it. And we will keep improving so don't be afraid that it will be enforced: if this does not work out we will discard it. P.S. Kudos to dcaro, most of the work was done by him, and most of this text too. +1 from me, releasing CI from running non critical and un-essential jobs will not only reduce load from ci, and shorted response time for developers, it will allow us to add much
Re: [ovirt-devel] gerrit+ci improvement proposal
- Original Message - From: Max Kovgan mkov...@redhat.com To: de...@ovirt.org Cc: infra@ovirt.org Sent: Wednesday, June 3, 2015 8:22:54 PM Subject: [ovirt-devel] gerrit+ci improvement proposal Hi everyone! We really want to have reliable and snappy CI: to allow short cycles and encourage developers to write tests. # Problem Many patches are neither ready for review nor for CI upon submission, which is OK. But running all the jobs on those patches with limited resources results in: overloaded resources, slow response time, unhappy developers. # Proposed Solution To run less jobs we know we don’t need to, thus making more resources for the jobs we need to run. We have been experimenting to make our CI stabler and quicker to respond by using gerrit flags. This has improved in both directions very well internally. Now it seems a good time to let all the oVirt projects to use this. This solution indirectly promotes reviews and quick tests - “to fail early”, yet full blown static code analysis and long tests to run “when ready”. # How it works 2 new gerrit independent flags are added to gerrit. ## CI flag Will express patch CI status. Values: * +1 CI passed * 0 CI did not run yet * -1 CI failed -1 You must have CI error state. Most of the errors I have seen, the CI fail to run, and the failure is not related to the tested patch. Permissions for setting: project maintainers (for special cases) should be able to set/override (except Jenkins). ## Workflow flag Will express patch “workflow” state. Values: * 0 Work In Progress * +1 Ready For Review * +2 Ready For Merge Permissions for setting: Owner can set +1, Project Maintainers can set +2 ## Review + CI Integration: Merging [“Submit” button to appear] will require: Review+1, CI+1, Workflow+2 Patch lifecycle now is: --- patch state |owner |reviewer |maintainer |CI tests |pass --- added/updated |- |-|- |quick|CI+1 review|Workflow+1|Review+1 |- |heavy |CI+1 merge ready |- |-|Workflow+2 |gating |CI+1 merge |- |-|merge |merge|CI+1 -1 Cannot require CI+1, the CI is not reliable enough yet. Even if the CI will be reliable, a failed test which is not related to the submitted patch should not block unrelated changes. Changes from current workflow: Owner only adds reviewers, now owner needs to set Workflow+1 for the patch to be reviewed, and heavily auto-tested. Maintainer now needs to set Workflow+2 and wait for Submit button to appear after CI has completed running gating tests. -1 This means more work for developers. Instead, make workflow default to +1. If the want to disable the CI, she can set it to 0. Developers need more power and less process overhead. Nir ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: [ovirt-devel] gerrit+ci improvement proposal
On 06/04, Nir Soffer wrote: - Original Message - From: Max Kovgan mkov...@redhat.com To: de...@ovirt.org Cc: infra@ovirt.org Sent: Wednesday, June 3, 2015 8:22:54 PM Subject: [ovirt-devel] gerrit+ci improvement proposal Hi everyone! We really want to have reliable and snappy CI: to allow short cycles and encourage developers to write tests. # Problem Many patches are neither ready for review nor for CI upon submission, which is OK. But running all the jobs on those patches with limited resources results in: overloaded resources, slow response time, unhappy developers. # Proposed Solution To run less jobs we know we don’t need to, thus making more resources for the jobs we need to run. We have been experimenting to make our CI stabler and quicker to respond by using gerrit flags. This has improved in both directions very well internally. Now it seems a good time to let all the oVirt projects to use this. This solution indirectly promotes reviews and quick tests - “to fail early”, yet full blown static code analysis and long tests to run “when ready”. # How it works 2 new gerrit independent flags are added to gerrit. ## CI flag Will express patch CI status. Values: * +1 CI passed * 0 CI did not run yet * -1 CI failed -1 You must have CI error state. Most of the errors I have seen, the CI fail to run, and the failure is not related to the tested patch. You'll have to prove this, as the last time we discussed it less than 10% of the failures from the previous 2 weeks were that case. Also that's why maintainers can set +1 here and -1 is not a blocker. Permissions for setting: project maintainers (for special cases) should be able to set/override (except Jenkins). ## Workflow flag Will express patch “workflow” state. Values: * 0 Work In Progress * +1 Ready For Review * +2 Ready For Merge Permissions for setting: Owner can set +1, Project Maintainers can set +2 ## Review + CI Integration: Merging [“Submit” button to appear] will require: Review+1, CI+1, Workflow+2 Patch lifecycle now is: --- patch state |owner |reviewer |maintainer |CI tests |pass --- added/updated |- |-|- |quick|CI+1 review|Workflow+1|Review+1 |- |heavy|CI+1 merge ready |- |-|Workflow+2 |gating |CI+1 merge |- |-|merge |merge|CI+1 -1 Cannot require CI+1, the CI is not reliable enough yet. Even if the CI will be reliable, a failed test which is not related to the submitted patch should not block unrelated changes. This situation makes no sense, if it's reliable theres no error due to non changed stuff (unless the issue that triggers the error is part of your patch history, because it's merged or you are based on a broken patch, in this case it should fail and block) Changes from current workflow: Owner only adds reviewers, now owner needs to set Workflow+1 for the patch to be reviewed, and heavily auto-tested. Maintainer now needs to set Workflow+2 and wait for Submit button to appear after CI has completed running gating tests. -1 This means more work for developers. Instead, make workflow default to +1. If the want to disable the CI, she can set it to 0. Developers need more power and less process overhead. We have to make some stats, but from the small sample of patches that we checked, most than half of them were not ready to pass any ci, it's a bit more work, if you keep using drafts, if you don't it's the same. It's not ment to be used alongside drafts but instead of it. So the only case where it increases the overhead is if you did not use drafts, in which case you should start using them, so no real downside there. Nir ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization RD Tel.: +420 532 294 605 Email: dc...@redhat.com Web: www.redhat.com RHT Global #: 82-62605 pgp4PxhcWDOyP.pgp Description: PGP signature ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Access to ovirt-srv08.ovirt.org
On Thu, 2015-06-04 at 04:50 -0400, Eyal Edri wrote: - Original Message - From: Tolik Litvosky tlito...@redhat.com To: Eyal Edri ee...@redhat.com Cc: infra@ovirt.org, de...@ovirt.org Sent: Thursday, June 4, 2015 11:44:09 AM Subject: Re: Access to ovirt-srv08.ovirt.org On Thu, 2015-06-04 at 02:31 -0400, Eyal Edri wrote: - Original Message - From: Tolik Litovsky tlito...@redhat.com To: ee...@redhat.com Cc: infra@ovirt.org Sent: Thursday, June 4, 2015 9:28:27 AM Subject: RE: Access to ovirt-srv08.ovirt.org The question is the nesting depth. We run currently in depth 2 . VM inside VM. Will depth of 3 will work. VM slave, and another 2 VM nesting above it. KVM people? Plz help. adding devel, but please elaborate on the depth, does it have to be nested or you're doing it because you only had 1 bare metal host till now (think you'll have 2 nested vms for example) The test requires depth 2 nested . We create an engine with hosts and storage. And on top of them we run another VM to test the ovirt features. This is the test scenario. OK, then we need to run a test and see the performance for it, because i think we will we gain many benefits from it, in terms of management and stability. Eyal , this testenv will most likely not be in my responsibility longer. But I know that there are several bare metals dedicated to this jobs. Can we plz bring them up , I belive it will be very helpfull before we finish the nested slaves investigation. Tolik. Original message From: Eyal Edri ee...@redhat.com Date: 03/06/2015 11:14 PM (GMT+02:00) To: tlito...@redhat.com Cc: infra@ovirt.org Subject: Re: Access to ovirt-srv08.ovirt.org I'd like to move the system testing framework to use nested vms instead of directly using the baremetal hosts. we did use a single host to POC the system, i think now that we're about to expand it will be much more managble and scalable if we'll start using nested VMs which will be managed via the hosted engine. e. - Original Message - From: Tolik Litvosky tlito...@redhat.com To: infra@ovirt.org Sent: Wednesday, June 3, 2015 3:59:36 PM Subject: Access to ovirt-srv08.ovirt.org Hi I need access to the ovirt-srv08.ovirt.org in order to update the testenv CI repo directory. -- Best regards Tolik Litovsky RHEV-H Team Red Hat Red Hat: trustworthy, transformative technology. Powered by the community. Connect at redhat.com ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra -- Best regards Tolik Litovsky RHEV-H Team Red Hat Red Hat: trustworthy, transformative technology. Powered by the community. Connect at redhat.com -- Best regards Tolik Litovsky RHEV-H Team Red Hat Red Hat: trustworthy, transformative technology. Powered by the community. Connect at redhat.com ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: [ovirt-devel] gerrit+ci improvement proposal
- Original Message - From: David Caro dcaro...@redhat.com To: Nir Soffer nsof...@redhat.com Cc: Oved Ourfali ov...@redhat.com, Eyal Edri ee...@redhat.com, infra@ovirt.org, de...@ovirt.org Sent: Thursday, June 4, 2015 11:34:10 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal On 06/04, Nir Soffer wrote: - Original Message - From: Oved Ourfali ov...@redhat.com To: Eyal Edri ee...@redhat.com Cc: de...@ovirt.org, infra@ovirt.org Sent: Thursday, June 4, 2015 10:03:02 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal - Original Message - From: Eyal Edri ee...@redhat.com To: Sandro Bonazzola sbona...@redhat.com Cc: infra@ovirt.org, de...@ovirt.org Sent: Thursday, June 4, 2015 9:46:40 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal - Original Message - From: Sandro Bonazzola sbona...@redhat.com To: Eyal Edri ee...@redhat.com, Max Kovgan mkov...@redhat.com Cc: de...@ovirt.org, infra@ovirt.org Sent: Thursday, June 4, 2015 9:11:10 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal Il 03/06/2015 21:46, Eyal Edri ha scritto: - Original Message - From: Max Kovgan mkov...@redhat.com To: de...@ovirt.org Cc: infra@ovirt.org Sent: Wednesday, June 3, 2015 8:22:54 PM Subject: [ovirt-devel] gerrit+ci improvement proposal Hi everyone! We really want to have reliable and snappy CI: to allow short cycles and encourage developers to write tests. # Problem Many patches are neither ready for review nor for CI upon submission, which is OK. But running all the jobs on those patches with limited resources results in: overloaded resources, slow response time, unhappy developers. # Proposed Solution To run less jobs we know we don’t need to, thus making more resources for the jobs we need to run. We have been experimenting to make our CI stabler and quicker to respond by using gerrit flags. This has improved in both directions very well internally. Now it seems a good time to let all the oVirt projects to use this. This solution indirectly promotes reviews and quick tests - “to fail early”, yet full blown static code analysis and long tests to run “when ready”. # How it works 2 new gerrit independent flags are added to gerrit. ## CI flag Will express patch CI status. Values: * +1 CI passed * 0 CI did not run yet * -1 CI failed Permissions for setting: project maintainers (for special cases) should be able to set/override (except Jenkins). ## Workflow flag Will express patch “workflow” state. Values: * 0 Work In Progress * +1 Ready For Review * +2 Ready For Merge Permissions for setting: Owner can set +1, Project Maintainers can set +2 ## Review + CI Integration: Merging [“Submit” button to appear] will require: Review+1, CI+1, Workflow+2 Patch lifecycle now is: --- patch state |owner |reviewer |maintainer |CI tests |pass --- added/updated |- |-|- |quick|CI+1 review|Workflow+1|Review+1 |- |heavy |CI+1 merge ready |- |-|Workflow+2 |gating |CI+1 merge |- |-|merge |merge|CI+1 Changes from current workflow: Owner only adds reviewers, now owner needs to set Workflow+1 for the patch to be reviewed, and heavily auto-tested. Maintainer now needs to set Workflow+2 and wait for Submit button to appear after CI has completed running gating tests. Next step will be to automate merge the change after Workflow+2 has been set by the Maintainer and gating tests passed. ## Why now? It is elimination of waste. The sooner - the better. The solution has been used for a while and it works. Resolving the problem without gerrit involved will lead to adding unreliable code into jobs, and will still be prone to problems: Just recently, 3d ago we’ve tried detecting what to run from jenkins relying only on gerrit comments so that upon Verified+1, we’d run the job. We could not use “Review+1”, because it makes no sense at all, so we left the job to set Verified+1. Meaning - re-trigger itself
Re: Access to ovirt-srv08.ovirt.org
- Original Message - From: David Caro dcaro...@redhat.com To: Tolik Litvosky tlito...@redhat.com Cc: Eyal Edri ee...@redhat.com, infra@ovirt.org, de...@ovirt.org Sent: Thursday, June 4, 2015 12:01:28 PM Subject: Re: Access to ovirt-srv08.ovirt.org On 06/04, Tolik Litvosky wrote: On Thu, 2015-06-04 at 04:50 -0400, Eyal Edri wrote: - Original Message - From: Tolik Litvosky tlito...@redhat.com To: Eyal Edri ee...@redhat.com Cc: infra@ovirt.org, de...@ovirt.org Sent: Thursday, June 4, 2015 11:44:09 AM Subject: Re: Access to ovirt-srv08.ovirt.org On Thu, 2015-06-04 at 02:31 -0400, Eyal Edri wrote: - Original Message - From: Tolik Litovsky tlito...@redhat.com To: ee...@redhat.com Cc: infra@ovirt.org Sent: Thursday, June 4, 2015 9:28:27 AM Subject: RE: Access to ovirt-srv08.ovirt.org The question is the nesting depth. We run currently in depth 2 . VM inside VM. Will depth of 3 will work. VM slave, and another 2 VM nesting above it. KVM people? Plz help. adding devel, but please elaborate on the depth, does it have to be nested or you're doing it because you only had 1 bare metal host till now (think you'll have 2 nested vms for example) The test requires depth 2 nested . We create an engine with hosts and storage. And on top of them we run another VM to test the ovirt features. This is the test scenario. OK, then we need to run a test and see the performance for it, because i think we will we gain many benefits from it, in terms of management and stability. Eyal , this testenv will most likely not be in my responsibility longer. But I know that there are several bare metals dedicated to this jobs. Can we plz bring them up , I belive it will be very helpfull before we finish the nested slaves investigation. There's only one afaik, and you are already using it. there are 6 more servers, but they are not ready yet, we will start working on them soon, i hope they will be ready in a week's time, we will update. regardless the system test maintainer should run the nested test so we'll know if we can leverage this. Tolik. Original message From: Eyal Edri ee...@redhat.com Date: 03/06/2015 11:14 PM (GMT+02:00) To: tlito...@redhat.com Cc: infra@ovirt.org Subject: Re: Access to ovirt-srv08.ovirt.org I'd like to move the system testing framework to use nested vms instead of directly using the baremetal hosts. we did use a single host to POC the system, i think now that we're about to expand it will be much more managble and scalable if we'll start using nested VMs which will be managed via the hosted engine. e. - Original Message - From: Tolik Litvosky tlito...@redhat.com To: infra@ovirt.org Sent: Wednesday, June 3, 2015 3:59:36 PM Subject: Access to ovirt-srv08.ovirt.org Hi I need access to the ovirt-srv08.ovirt.org in order to update the testenv CI repo directory. -- Best regards Tolik Litovsky RHEV-H Team Red Hat Red Hat: trustworthy, transformative technology. Powered by the community. Connect at redhat.com ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra -- Best regards Tolik Litovsky RHEV-H Team Red Hat Red Hat: trustworthy, transformative technology. Powered by the community. Connect at redhat.com -- Best regards Tolik Litovsky RHEV-H Team Red Hat Red Hat: trustworthy, transformative technology. Powered by the community. Connect at redhat.com ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization RD Tel.: +420 532 294 605 Email: dc...@redhat.com Web: www.redhat.com RHT Global #: 82-62605 ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Access to ovirt-srv08.ovirt.org
On Thu, 2015-06-04 at 05:08 -0400, Eyal Edri wrote:
- Original Message -
From: David Caro dcaro...@redhat.com
To: Tolik Litvosky tlito...@redhat.com
Cc: Eyal Edri ee...@redhat.com, infra@ovirt.org,
de...@ovirt.org
Sent: Thursday, June 4, 2015 12:01:28 PM
Subject: Re: Access to ovirt-srv08.ovirt.org
On 06/04, Tolik Litvosky wrote:
On Thu, 2015-06-04 at 04:50 -0400, Eyal Edri wrote:
- Original Message -
From: Tolik Litvosky tlito...@redhat.com
To: Eyal Edri ee...@redhat.com
Cc: infra@ovirt.org, de...@ovirt.org
Sent: Thursday, June 4, 2015 11:44:09 AM
Subject: Re: Access to ovirt-srv08.ovirt.org
On Thu, 2015-06-04 at 02:31 -0400, Eyal Edri wrote:
- Original Message -
From: Tolik Litovsky tlito...@redhat.com
To: ee...@redhat.com
Cc: infra@ovirt.org
Sent: Thursday, June 4, 2015 9:28:27 AM
Subject: RE: Access to ovirt-srv08.ovirt.org
The question is the nesting depth.
We run currently in depth 2 .
VM inside VM.
Will depth of 3 will work.
VM slave, and another 2 VM nesting above it.
KVM people? Plz help.
adding devel, but please elaborate on the depth,
does it have to be nested or you're doing it because
you only had 1 bare metal host till now (think you'll have
2
nested
vms for example)
The test requires depth 2 nested . We create an engine with
hosts
and
storage. And on top of them we run another VM to test the
ovirt
features.
This is the test scenario.
OK, then we need to run a test and see the performance for it,
because i think
we will we gain many benefits from it, in terms of management
and
stability.
Eyal , this testenv will most likely not be in my responsibility
longer.
But I know that there are several bare metals dedicated to this
jobs.
Can we plz bring them up , I belive it will be very helpfull
before we
finish the nested slaves investigation.
There's only one afaik, and you are already using it.
there are 6 more servers, but they are not ready yet,
we will start working on them soon, i hope they will be ready
in a week's time, we will update.
regardless the system test maintainer should run the nested test so
we'll know if we can leverage this.
You can add a nested host to the group and we can try.
I am for sure not doing it on my laptop or minidell.
It will not give any reasonable estimate.
Tolik.
Original message
From: Eyal Edri ee...@redhat.com
Date: 03/06/2015 11:14 PM (GMT+02:00)
To: tlito...@redhat.com
Cc: infra@ovirt.org
Subject: Re: Access to ovirt-srv08.ovirt.org
I'd like to move the system testing framework to use
nested vms instead of directly using the baremetal hosts.
we did use a single host to POC the system, i think now
that
we're
about to expand it will be much more managble and
scalable if
we'll
start using nested VMs which will be managed via the
hosted
engine.
e.
- Original Message -
From: Tolik Litvosky tlito...@redhat.com
To: infra@ovirt.org
Sent: Wednesday, June 3, 2015 3:59:36 PM
Subject: Access to ovirt-srv08.ovirt.org
Hi
I need access to the ovirt-srv08.ovirt.org in order to
update
the
testenv CI repo directory.
--
Best regards
Tolik Litovsky
RHEV-H Team
Red Hat
Red Hat: trustworthy, transformative technology.
Powered by
the
community.
Connect at redhat.com
___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
--
Best regards
Tolik Litovsky
RHEV-H Team
Red Hat
Red Hat: trustworthy, transformative technology. Powered by
the
community.
Connect at redhat.com
--
Best regards
Tolik Litovsky
RHEV-H Team
Red Hat
Red Hat: trustworthy, transformative technology. Powered by the
community.
Connect at redhat.com
___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
--
David Caro
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization RD
Tel.: +420 532 294 605
Email: dc...@redhat.com
Web: www.redhat.com
RHT Global #: 82-62605
___
Infra mailing list
Infra@ovirt.org
Re: Access to ovirt-srv08.ovirt.org
On 06/04, Eyal Edri wrote:
- Original Message -
From: David Caro dcaro...@redhat.com
To: Tolik Litvosky tlito...@redhat.com
Cc: Eyal Edri ee...@redhat.com, infra@ovirt.org, de...@ovirt.org
Sent: Thursday, June 4, 2015 12:01:28 PM
Subject: Re: Access to ovirt-srv08.ovirt.org
On 06/04, Tolik Litvosky wrote:
On Thu, 2015-06-04 at 04:50 -0400, Eyal Edri wrote:
- Original Message -
From: Tolik Litvosky tlito...@redhat.com
To: Eyal Edri ee...@redhat.com
Cc: infra@ovirt.org, de...@ovirt.org
Sent: Thursday, June 4, 2015 11:44:09 AM
Subject: Re: Access to ovirt-srv08.ovirt.org
On Thu, 2015-06-04 at 02:31 -0400, Eyal Edri wrote:
- Original Message -
From: Tolik Litovsky tlito...@redhat.com
To: ee...@redhat.com
Cc: infra@ovirt.org
Sent: Thursday, June 4, 2015 9:28:27 AM
Subject: RE: Access to ovirt-srv08.ovirt.org
The question is the nesting depth.
We run currently in depth 2 .
VM inside VM.
Will depth of 3 will work.
VM slave, and another 2 VM nesting above it.
KVM people? Plz help.
adding devel, but please elaborate on the depth,
does it have to be nested or you're doing it because
you only had 1 bare metal host till now (think you'll have 2
nested
vms for example)
The test requires depth 2 nested . We create an engine with hosts
and
storage. And on top of them we run another VM to test the ovirt
features.
This is the test scenario.
OK, then we need to run a test and see the performance for it,
because i think
we will we gain many benefits from it, in terms of management and
stability.
Eyal , this testenv will most likely not be in my responsibility
longer.
But I know that there are several bare metals dedicated to this jobs.
Can we plz bring them up , I belive it will be very helpfull before we
finish the nested slaves investigation.
There's only one afaik, and you are already using it.
there are 6 more servers, but they are not ready yet,
we will start working on them soon, i hope they will be ready
in a week's time, we will update.
regardless the system test maintainer should run the nested test so
we'll know if we can leverage this.
I was not aware they were dedicated for those jobs
Tolik.
Original message
From: Eyal Edri ee...@redhat.com
Date: 03/06/2015 11:14 PM (GMT+02:00)
To: tlito...@redhat.com
Cc: infra@ovirt.org
Subject: Re: Access to ovirt-srv08.ovirt.org
I'd like to move the system testing framework to use
nested vms instead of directly using the baremetal hosts.
we did use a single host to POC the system, i think now that
we're
about to expand it will be much more managble and scalable if
we'll
start using nested VMs which will be managed via the hosted
engine.
e.
- Original Message -
From: Tolik Litvosky tlito...@redhat.com
To: infra@ovirt.org
Sent: Wednesday, June 3, 2015 3:59:36 PM
Subject: Access to ovirt-srv08.ovirt.org
Hi
I need access to the ovirt-srv08.ovirt.org in order to update
the
testenv CI repo directory.
--
Best regards
Tolik Litovsky
RHEV-H Team
Red Hat
Red Hat: trustworthy, transformative technology. Powered by
the
community.
Connect at redhat.com
___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
--
Best regards
Tolik Litovsky
RHEV-H Team
Red Hat
Red Hat: trustworthy, transformative technology. Powered by the
community.
Connect at redhat.com
--
Best regards
Tolik Litovsky
RHEV-H Team
Red Hat
Red Hat: trustworthy, transformative technology. Powered by the
community.
Connect at redhat.com
___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
--
David Caro
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization RD
Tel.: +420 532 294 605
Email: dc...@redhat.com
Web: www.redhat.com
RHT Global #: 82-62605
___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
--
David Caro
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization RD
Tel.: +420 532 294 605
Email: dc...@redhat.com
Web: www.redhat.com
RHT
Re: [ovirt-devel] gerrit+ci improvement proposal
- Original Message - From: Nir Soffer nsof...@redhat.com To: David Caro dcaro...@redhat.com Cc: de...@ovirt.org, infra@ovirt.org Sent: Thursday, June 4, 2015 10:57:30 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal - Original Message - From: David Caro dcaro...@redhat.com To: Nir Soffer nsof...@redhat.com Cc: Max Kovgan mkov...@redhat.com, infra@ovirt.org, de...@ovirt.org Sent: Thursday, June 4, 2015 10:35:12 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal On 06/04, Nir Soffer wrote: - Original Message - From: Max Kovgan mkov...@redhat.com To: de...@ovirt.org Cc: infra@ovirt.org Sent: Wednesday, June 3, 2015 8:22:54 PM Subject: [ovirt-devel] gerrit+ci improvement proposal Hi everyone! We really want to have reliable and snappy CI: to allow short cycles and encourage developers to write tests. # Problem Many patches are neither ready for review nor for CI upon submission, which is OK. But running all the jobs on those patches with limited resources results in: overloaded resources, slow response time, unhappy developers. # Proposed Solution To run less jobs we know we don’t need to, thus making more resources for the jobs we need to run. We have been experimenting to make our CI stabler and quicker to respond by using gerrit flags. This has improved in both directions very well internally. Now it seems a good time to let all the oVirt projects to use this. This solution indirectly promotes reviews and quick tests - “to fail early”, yet full blown static code analysis and long tests to run “when ready”. # How it works 2 new gerrit independent flags are added to gerrit. ## CI flag Will express patch CI status. Values: * +1 CI passed * 0 CI did not run yet * -1 CI failed -1 You must have CI error state. Most of the errors I have seen, the CI fail to run, and the failure is not related to the tested patch. You'll have to prove this, as the last time we discussed it less than 10% of the failures from the previous 2 weeks were that case. 10% is 10% too much :-) Also that's why maintainers can set +1 here and -1 is not a blocker. Ok, than its fine. Permissions for setting: project maintainers (for special cases) should be able to set/override (except Jenkins). ## Workflow flag Will express patch “workflow” state. Values: * 0 Work In Progress * +1 Ready For Review * +2 Ready For Merge Permissions for setting: Owner can set +1, Project Maintainers can set +2 ## Review + CI Integration: Merging [“Submit” button to appear] will require: Review+1, CI+1, Workflow+2 Patch lifecycle now is: --- patch state |owner |reviewer |maintainer |CI tests |pass --- added/updated |- |-|- |quick|CI+1 review|Workflow+1|Review+1 |- |heavy|CI+1 merge ready |- |-|Workflow+2 |gating |CI+1 merge |- |-|merge |merge|CI+1 -1 Cannot require CI+1, the CI is not reliable enough yet. Even if the CI will be reliable, a failed test which is not related to the submitted patch should not block unrelated changes. This situation makes no sense, if it's reliable theres no error due to non changed stuff (unless the issue that triggers the error is part of your patch history, because it's merged or you are based on a broken patch, in this case it should fail and block) Here is an example: http://jenkins.ovirt.org/job/vdsm_master_unit-tests_created/3314/ Failed due to flaky networking tests, the failure is not related to the patch, which changes multipath configuration for some device. Flaky tests should be fixed, but blocking development because of them is not productive. The purpose of the CI is to help us move faster, not slow us down. The pep8 tests is doing this right - it checks only the code added in the last patch. In the current situation, the only way to move forward is to fix the flaky tests or mark them as @brokentest. Changes from current workflow: Owner only adds reviewers, now owner needs to set Workflow+1 for the patch to be reviewed, and heavily auto-tested. Maintainer now needs to set Workflow+2 and wait for Submit button to appear after CI has completed running gating tests. -1 This means more work for developers. Instead, make workflow default to +1. If the want to disable the CI, she can set it to 0. Developers need more power and less process overhead.
Re: [ovirt-devel] gerrit+ci improvement proposal
On 06/04, Nir Soffer wrote: - Original Message - From: Oved Ourfali ov...@redhat.com To: Eyal Edri ee...@redhat.com Cc: de...@ovirt.org, infra@ovirt.org Sent: Thursday, June 4, 2015 10:03:02 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal - Original Message - From: Eyal Edri ee...@redhat.com To: Sandro Bonazzola sbona...@redhat.com Cc: infra@ovirt.org, de...@ovirt.org Sent: Thursday, June 4, 2015 9:46:40 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal - Original Message - From: Sandro Bonazzola sbona...@redhat.com To: Eyal Edri ee...@redhat.com, Max Kovgan mkov...@redhat.com Cc: de...@ovirt.org, infra@ovirt.org Sent: Thursday, June 4, 2015 9:11:10 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal Il 03/06/2015 21:46, Eyal Edri ha scritto: - Original Message - From: Max Kovgan mkov...@redhat.com To: de...@ovirt.org Cc: infra@ovirt.org Sent: Wednesday, June 3, 2015 8:22:54 PM Subject: [ovirt-devel] gerrit+ci improvement proposal Hi everyone! We really want to have reliable and snappy CI: to allow short cycles and encourage developers to write tests. # Problem Many patches are neither ready for review nor for CI upon submission, which is OK. But running all the jobs on those patches with limited resources results in: overloaded resources, slow response time, unhappy developers. # Proposed Solution To run less jobs we know we don’t need to, thus making more resources for the jobs we need to run. We have been experimenting to make our CI stabler and quicker to respond by using gerrit flags. This has improved in both directions very well internally. Now it seems a good time to let all the oVirt projects to use this. This solution indirectly promotes reviews and quick tests - “to fail early”, yet full blown static code analysis and long tests to run “when ready”. # How it works 2 new gerrit independent flags are added to gerrit. ## CI flag Will express patch CI status. Values: * +1 CI passed * 0 CI did not run yet * -1 CI failed Permissions for setting: project maintainers (for special cases) should be able to set/override (except Jenkins). ## Workflow flag Will express patch “workflow” state. Values: * 0 Work In Progress * +1 Ready For Review * +2 Ready For Merge Permissions for setting: Owner can set +1, Project Maintainers can set +2 ## Review + CI Integration: Merging [“Submit” button to appear] will require: Review+1, CI+1, Workflow+2 Patch lifecycle now is: --- patch state |owner |reviewer |maintainer |CI tests |pass --- added/updated |- |-|- |quick|CI+1 review|Workflow+1|Review+1 |- |heavy |CI+1 merge ready |- |-|Workflow+2 |gating |CI+1 merge |- |-|merge |merge|CI+1 Changes from current workflow: Owner only adds reviewers, now owner needs to set Workflow+1 for the patch to be reviewed, and heavily auto-tested. Maintainer now needs to set Workflow+2 and wait for Submit button to appear after CI has completed running gating tests. Next step will be to automate merge the change after Workflow+2 has been set by the Maintainer and gating tests passed. ## Why now? It is elimination of waste. The sooner - the better. The solution has been used for a while and it works. Resolving the problem without gerrit involved will lead to adding unreliable code into jobs, and will still be prone to problems: Just recently, 3d ago we’ve tried detecting what to run from jenkins relying only on gerrit comments so that upon Verified+1, we’d run the job. We could not use “Review+1”, because it makes no sense at all, so we left the job to set Verified+1. Meaning - re-trigger itself immediately more than 1 times. Jenkins and its visitors very unhappy, and we had to stop those jobs, clean up the queue, and spam developers. ## OK OK OK. Now what? Now we want your comments and opinions before pushing this further: Please participate in this thread, so we can start trying it out. Ask, Suggest better ideas, all this is welcome. Best Regards! N.B. Of course, this is not written
Re: [ovirt-devel] gerrit+ci improvement proposal
On 06/04, Nir Soffer wrote: - Original Message - From: David Caro dcaro...@redhat.com To: Nir Soffer nsof...@redhat.com Cc: Oved Ourfali ov...@redhat.com, Eyal Edri ee...@redhat.com, infra@ovirt.org, de...@ovirt.org Sent: Thursday, June 4, 2015 11:34:10 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal On 06/04, Nir Soffer wrote: - Original Message - From: Oved Ourfali ov...@redhat.com To: Eyal Edri ee...@redhat.com Cc: de...@ovirt.org, infra@ovirt.org Sent: Thursday, June 4, 2015 10:03:02 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal - Original Message - From: Eyal Edri ee...@redhat.com To: Sandro Bonazzola sbona...@redhat.com Cc: infra@ovirt.org, de...@ovirt.org Sent: Thursday, June 4, 2015 9:46:40 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal - Original Message - From: Sandro Bonazzola sbona...@redhat.com To: Eyal Edri ee...@redhat.com, Max Kovgan mkov...@redhat.com Cc: de...@ovirt.org, infra@ovirt.org Sent: Thursday, June 4, 2015 9:11:10 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal Il 03/06/2015 21:46, Eyal Edri ha scritto: - Original Message - From: Max Kovgan mkov...@redhat.com To: de...@ovirt.org Cc: infra@ovirt.org Sent: Wednesday, June 3, 2015 8:22:54 PM Subject: [ovirt-devel] gerrit+ci improvement proposal Hi everyone! We really want to have reliable and snappy CI: to allow short cycles and encourage developers to write tests. # Problem Many patches are neither ready for review nor for CI upon submission, which is OK. But running all the jobs on those patches with limited resources results in: overloaded resources, slow response time, unhappy developers. # Proposed Solution To run less jobs we know we don’t need to, thus making more resources for the jobs we need to run. We have been experimenting to make our CI stabler and quicker to respond by using gerrit flags. This has improved in both directions very well internally. Now it seems a good time to let all the oVirt projects to use this. This solution indirectly promotes reviews and quick tests - “to fail early”, yet full blown static code analysis and long tests to run “when ready”. # How it works 2 new gerrit independent flags are added to gerrit. ## CI flag Will express patch CI status. Values: * +1 CI passed * 0 CI did not run yet * -1 CI failed Permissions for setting: project maintainers (for special cases) should be able to set/override (except Jenkins). ## Workflow flag Will express patch “workflow” state. Values: * 0 Work In Progress * +1 Ready For Review * +2 Ready For Merge Permissions for setting: Owner can set +1, Project Maintainers can set +2 ## Review + CI Integration: Merging [“Submit” button to appear] will require: Review+1, CI+1, Workflow+2 Patch lifecycle now is: --- patch state |owner |reviewer |maintainer |CI tests |pass --- added/updated |- |-|- |quick|CI+1 review|Workflow+1|Review+1 |- |heavy |CI+1 merge ready |- |-|Workflow+2 |gating |CI+1 merge |- |-|merge |merge|CI+1 Changes from current workflow: Owner only adds reviewers, now owner needs to set Workflow+1 for the patch to be reviewed, and heavily auto-tested. Maintainer now needs to set Workflow+2 and wait for Submit button to appear after CI has completed running gating tests. Next step will be to automate merge the change after Workflow+2 has been set by the Maintainer and gating tests passed. ## Why now? It is elimination of waste. The sooner - the better. The solution has been used for a while and it works. Resolving the problem without gerrit involved will lead to adding unreliable code into jobs, and will still be prone to problems: Just recently, 3d ago we’ve tried detecting what to run from jenkins relying only on gerrit comments so that upon Verified+1, we’d
Re: [ovirt-devel] gerrit+ci improvement proposal
- Original Message - From: David Caro dcaro...@redhat.com To: Nir Soffer nsof...@redhat.com Cc: Max Kovgan mkov...@redhat.com, infra@ovirt.org, de...@ovirt.org Sent: Thursday, June 4, 2015 10:35:12 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal On 06/04, Nir Soffer wrote: - Original Message - From: Max Kovgan mkov...@redhat.com To: de...@ovirt.org Cc: infra@ovirt.org Sent: Wednesday, June 3, 2015 8:22:54 PM Subject: [ovirt-devel] gerrit+ci improvement proposal Hi everyone! We really want to have reliable and snappy CI: to allow short cycles and encourage developers to write tests. # Problem Many patches are neither ready for review nor for CI upon submission, which is OK. But running all the jobs on those patches with limited resources results in: overloaded resources, slow response time, unhappy developers. # Proposed Solution To run less jobs we know we don’t need to, thus making more resources for the jobs we need to run. We have been experimenting to make our CI stabler and quicker to respond by using gerrit flags. This has improved in both directions very well internally. Now it seems a good time to let all the oVirt projects to use this. This solution indirectly promotes reviews and quick tests - “to fail early”, yet full blown static code analysis and long tests to run “when ready”. # How it works 2 new gerrit independent flags are added to gerrit. ## CI flag Will express patch CI status. Values: * +1 CI passed * 0 CI did not run yet * -1 CI failed -1 You must have CI error state. Most of the errors I have seen, the CI fail to run, and the failure is not related to the tested patch. You'll have to prove this, as the last time we discussed it less than 10% of the failures from the previous 2 weeks were that case. 10% is 10% too much :-) Also that's why maintainers can set +1 here and -1 is not a blocker. Ok, than its fine. Permissions for setting: project maintainers (for special cases) should be able to set/override (except Jenkins). ## Workflow flag Will express patch “workflow” state. Values: * 0 Work In Progress * +1 Ready For Review * +2 Ready For Merge Permissions for setting: Owner can set +1, Project Maintainers can set +2 ## Review + CI Integration: Merging [“Submit” button to appear] will require: Review+1, CI+1, Workflow+2 Patch lifecycle now is: --- patch state |owner |reviewer |maintainer |CI tests |pass --- added/updated |- |-|- |quick|CI+1 review|Workflow+1|Review+1 |- |heavy |CI+1 merge ready |- |-|Workflow+2 |gating |CI+1 merge |- |-|merge |merge|CI+1 -1 Cannot require CI+1, the CI is not reliable enough yet. Even if the CI will be reliable, a failed test which is not related to the submitted patch should not block unrelated changes. This situation makes no sense, if it's reliable theres no error due to non changed stuff (unless the issue that triggers the error is part of your patch history, because it's merged or you are based on a broken patch, in this case it should fail and block) Here is an example: http://jenkins.ovirt.org/job/vdsm_master_unit-tests_created/3314/ Failed due to flaky networking tests, the failure is not related to the patch, which changes multipath configuration for some device. Flaky tests should be fixed, but blocking development because of them is not productive. The purpose of the CI is to help us move faster, not slow us down. The pep8 tests is doing this right - it checks only the code added in the last patch. In the current situation, the only way to move forward is to fix the flaky tests or mark them as @brokentest. Changes from current workflow: Owner only adds reviewers, now owner needs to set Workflow+1 for the patch to be reviewed, and heavily auto-tested. Maintainer now needs to set Workflow+2 and wait for Submit button to appear after CI has completed running gating tests. -1 This means more work for developers. Instead, make workflow default to +1. If the want to disable the CI, she can set it to 0. Developers need more power and less process overhead. We have to make some stats, but from the small sample of patches that we checked, most than half of them were not ready to pass any ci, Of course they were not ready, it is expected and good, and save developer time. The best way to check if a patch is ready for the CI is to let the CI run it :-) it's a bit more work, if you keep using drafts, if you don't it's the same. It's not ment to be used alongside
Re: Access to ovirt-srv08.ovirt.org
- Original Message -
From: Tolik Litvosky tlito...@redhat.com
To: Eyal Edri ee...@redhat.com
Cc: infra@ovirt.org, de...@ovirt.org
Sent: Thursday, June 4, 2015 11:44:09 AM
Subject: Re: Access to ovirt-srv08.ovirt.org
On Thu, 2015-06-04 at 02:31 -0400, Eyal Edri wrote:
- Original Message -
From: Tolik Litovsky tlito...@redhat.com
To: ee...@redhat.com
Cc: infra@ovirt.org
Sent: Thursday, June 4, 2015 9:28:27 AM
Subject: RE: Access to ovirt-srv08.ovirt.org
The question is the nesting depth.
We run currently in depth 2 .
VM inside VM.
Will depth of 3 will work.
VM slave, and another 2 VM nesting above it.
KVM people? Plz help.
adding devel, but please elaborate on the depth,
does it have to be nested or you're doing it because
you only had 1 bare metal host till now (think you'll have 2 nested
vms for example)
The test requires depth 2 nested . We create an engine with hosts and
storage. And on top of them we run another VM to test the ovirt features.
This is the test scenario.
OK, then we need to run a test and see the performance for it, because i think
we will we gain many benefits from it, in terms of management and stability.
Tolik.
Original message
From: Eyal Edri ee...@redhat.com
Date: 03/06/2015 11:14 PM (GMT+02:00)
To: tlito...@redhat.com
Cc: infra@ovirt.org
Subject: Re: Access to ovirt-srv08.ovirt.org
I'd like to move the system testing framework to use
nested vms instead of directly using the baremetal hosts.
we did use a single host to POC the system, i think now that we're
about to expand it will be much more managble and scalable if we'll
start using nested VMs which will be managed via the hosted engine.
e.
- Original Message -
From: Tolik Litvosky tlito...@redhat.com
To: infra@ovirt.org
Sent: Wednesday, June 3, 2015 3:59:36 PM
Subject: Access to ovirt-srv08.ovirt.org
Hi
I need access to the ovirt-srv08.ovirt.org in order to update the
testenv CI repo directory.
--
Best regards
Tolik Litovsky
RHEV-H Team
Red Hat
Red Hat: trustworthy, transformative technology. Powered by the
community.
Connect at redhat.com
___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
--
Best regards
Tolik Litovsky
RHEV-H Team
Red Hat
Red Hat: trustworthy, transformative technology. Powered by the
community.
Connect at redhat.com
___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
Re: [ovirt-devel] gerrit+ci improvement proposal
- Original Message - From: Oved Ourfali ov...@redhat.com To: Eyal Edri ee...@redhat.com Cc: de...@ovirt.org, infra@ovirt.org Sent: Thursday, June 4, 2015 10:03:02 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal - Original Message - From: Eyal Edri ee...@redhat.com To: Sandro Bonazzola sbona...@redhat.com Cc: infra@ovirt.org, de...@ovirt.org Sent: Thursday, June 4, 2015 9:46:40 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal - Original Message - From: Sandro Bonazzola sbona...@redhat.com To: Eyal Edri ee...@redhat.com, Max Kovgan mkov...@redhat.com Cc: de...@ovirt.org, infra@ovirt.org Sent: Thursday, June 4, 2015 9:11:10 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal Il 03/06/2015 21:46, Eyal Edri ha scritto: - Original Message - From: Max Kovgan mkov...@redhat.com To: de...@ovirt.org Cc: infra@ovirt.org Sent: Wednesday, June 3, 2015 8:22:54 PM Subject: [ovirt-devel] gerrit+ci improvement proposal Hi everyone! We really want to have reliable and snappy CI: to allow short cycles and encourage developers to write tests. # Problem Many patches are neither ready for review nor for CI upon submission, which is OK. But running all the jobs on those patches with limited resources results in: overloaded resources, slow response time, unhappy developers. # Proposed Solution To run less jobs we know we don’t need to, thus making more resources for the jobs we need to run. We have been experimenting to make our CI stabler and quicker to respond by using gerrit flags. This has improved in both directions very well internally. Now it seems a good time to let all the oVirt projects to use this. This solution indirectly promotes reviews and quick tests - “to fail early”, yet full blown static code analysis and long tests to run “when ready”. # How it works 2 new gerrit independent flags are added to gerrit. ## CI flag Will express patch CI status. Values: * +1 CI passed * 0 CI did not run yet * -1 CI failed Permissions for setting: project maintainers (for special cases) should be able to set/override (except Jenkins). ## Workflow flag Will express patch “workflow” state. Values: * 0 Work In Progress * +1 Ready For Review * +2 Ready For Merge Permissions for setting: Owner can set +1, Project Maintainers can set +2 ## Review + CI Integration: Merging [“Submit” button to appear] will require: Review+1, CI+1, Workflow+2 Patch lifecycle now is: --- patch state |owner |reviewer |maintainer |CI tests |pass --- added/updated |- |-|- |quick|CI+1 review|Workflow+1|Review+1 |- |heavy |CI+1 merge ready |- |-|Workflow+2 |gating |CI+1 merge |- |-|merge |merge|CI+1 Changes from current workflow: Owner only adds reviewers, now owner needs to set Workflow+1 for the patch to be reviewed, and heavily auto-tested. Maintainer now needs to set Workflow+2 and wait for Submit button to appear after CI has completed running gating tests. Next step will be to automate merge the change after Workflow+2 has been set by the Maintainer and gating tests passed. ## Why now? It is elimination of waste. The sooner - the better. The solution has been used for a while and it works. Resolving the problem without gerrit involved will lead to adding unreliable code into jobs, and will still be prone to problems: Just recently, 3d ago we’ve tried detecting what to run from jenkins relying only on gerrit comments so that upon Verified+1, we’d run the job. We could not use “Review+1”, because it makes no sense at all, so we left the job to set Verified+1. Meaning - re-trigger itself immediately more than 1 times. Jenkins and its visitors very unhappy, and we had to stop those jobs, clean up the queue, and spam developers. ## OK OK OK. Now what? Now we want your comments and opinions before pushing this further: Please participate in this thread, so we can start trying it out. Ask, Suggest better ideas, all this is welcome. Best Regards! N.B. Of course, this is not written in stone, in case we find a better approach on solving those issues, we will change to it. And we will keep improving so don't be afraid that it will be enforced: if this does not work
Re: Access to ovirt-srv08.ovirt.org
On Thu, 2015-06-04 at 02:31 -0400, Eyal Edri wrote: - Original Message - From: Tolik Litovsky tlito...@redhat.com To: ee...@redhat.com Cc: infra@ovirt.org Sent: Thursday, June 4, 2015 9:28:27 AM Subject: RE: Access to ovirt-srv08.ovirt.org The question is the nesting depth. We run currently in depth 2 . VM inside VM. Will depth of 3 will work. VM slave, and another 2 VM nesting above it. KVM people? Plz help. adding devel, but please elaborate on the depth, does it have to be nested or you're doing it because you only had 1 bare metal host till now (think you'll have 2 nested vms for example) The test requires depth 2 nested . We create an engine with hosts and storage. And on top of them we run another VM to test the ovirt features. This is the test scenario. Tolik. Original message From: Eyal Edri ee...@redhat.com Date: 03/06/2015 11:14 PM (GMT+02:00) To: tlito...@redhat.com Cc: infra@ovirt.org Subject: Re: Access to ovirt-srv08.ovirt.org I'd like to move the system testing framework to use nested vms instead of directly using the baremetal hosts. we did use a single host to POC the system, i think now that we're about to expand it will be much more managble and scalable if we'll start using nested VMs which will be managed via the hosted engine. e. - Original Message - From: Tolik Litvosky tlito...@redhat.com To: infra@ovirt.org Sent: Wednesday, June 3, 2015 3:59:36 PM Subject: Access to ovirt-srv08.ovirt.org Hi I need access to the ovirt-srv08.ovirt.org in order to update the testenv CI repo directory. -- Best regards Tolik Litovsky RHEV-H Team Red Hat Red Hat: trustworthy, transformative technology. Powered by the community. Connect at redhat.com ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra -- Best regards Tolik Litovsky RHEV-H Team Red Hat Red Hat: trustworthy, transformative technology. Powered by the community. Connect at redhat.com ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Access to ovirt-srv08.ovirt.org
On 06/04, Tolik Litvosky wrote: On Thu, 2015-06-04 at 04:50 -0400, Eyal Edri wrote: - Original Message - From: Tolik Litvosky tlito...@redhat.com To: Eyal Edri ee...@redhat.com Cc: infra@ovirt.org, de...@ovirt.org Sent: Thursday, June 4, 2015 11:44:09 AM Subject: Re: Access to ovirt-srv08.ovirt.org On Thu, 2015-06-04 at 02:31 -0400, Eyal Edri wrote: - Original Message - From: Tolik Litovsky tlito...@redhat.com To: ee...@redhat.com Cc: infra@ovirt.org Sent: Thursday, June 4, 2015 9:28:27 AM Subject: RE: Access to ovirt-srv08.ovirt.org The question is the nesting depth. We run currently in depth 2 . VM inside VM. Will depth of 3 will work. VM slave, and another 2 VM nesting above it. KVM people? Plz help. adding devel, but please elaborate on the depth, does it have to be nested or you're doing it because you only had 1 bare metal host till now (think you'll have 2 nested vms for example) The test requires depth 2 nested . We create an engine with hosts and storage. And on top of them we run another VM to test the ovirt features. This is the test scenario. OK, then we need to run a test and see the performance for it, because i think we will we gain many benefits from it, in terms of management and stability. Eyal , this testenv will most likely not be in my responsibility longer. But I know that there are several bare metals dedicated to this jobs. Can we plz bring them up , I belive it will be very helpfull before we finish the nested slaves investigation. There's only one afaik, and you are already using it. Tolik. Original message From: Eyal Edri ee...@redhat.com Date: 03/06/2015 11:14 PM (GMT+02:00) To: tlito...@redhat.com Cc: infra@ovirt.org Subject: Re: Access to ovirt-srv08.ovirt.org I'd like to move the system testing framework to use nested vms instead of directly using the baremetal hosts. we did use a single host to POC the system, i think now that we're about to expand it will be much more managble and scalable if we'll start using nested VMs which will be managed via the hosted engine. e. - Original Message - From: Tolik Litvosky tlito...@redhat.com To: infra@ovirt.org Sent: Wednesday, June 3, 2015 3:59:36 PM Subject: Access to ovirt-srv08.ovirt.org Hi I need access to the ovirt-srv08.ovirt.org in order to update the testenv CI repo directory. -- Best regards Tolik Litovsky RHEV-H Team Red Hat Red Hat: trustworthy, transformative technology. Powered by the community. Connect at redhat.com ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra -- Best regards Tolik Litovsky RHEV-H Team Red Hat Red Hat: trustworthy, transformative technology. Powered by the community. Connect at redhat.com -- Best regards Tolik Litovsky RHEV-H Team Red Hat Red Hat: trustworthy, transformative technology. Powered by the community. Connect at redhat.com ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization RD Tel.: +420 532 294 605 Email: dc...@redhat.com Web: www.redhat.com RHT Global #: 82-62605 pgp9ZURw8ChDp.pgp Description: PGP signature ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Access to ovirt-srv08.ovirt.org
- Original Message - From: David Caro dcaro...@redhat.com To: Eyal Edri ee...@redhat.com Cc: Dima Kuznetsov dkuzn...@redhat.com, Tolik Litvosky tlito...@redhat.com, infra@ovirt.org, de...@ovirt.org Sent: Thursday, June 4, 2015 12:11:05 PM Subject: Re: Access to ovirt-srv08.ovirt.org On 06/04, Eyal Edri wrote: - Original Message - From: David Caro dcaro...@redhat.com To: Tolik Litvosky tlito...@redhat.com Cc: Eyal Edri ee...@redhat.com, infra@ovirt.org, de...@ovirt.org Sent: Thursday, June 4, 2015 12:01:28 PM Subject: Re: Access to ovirt-srv08.ovirt.org On 06/04, Tolik Litvosky wrote: On Thu, 2015-06-04 at 04:50 -0400, Eyal Edri wrote: - Original Message - From: Tolik Litvosky tlito...@redhat.com To: Eyal Edri ee...@redhat.com Cc: infra@ovirt.org, de...@ovirt.org Sent: Thursday, June 4, 2015 11:44:09 AM Subject: Re: Access to ovirt-srv08.ovirt.org On Thu, 2015-06-04 at 02:31 -0400, Eyal Edri wrote: - Original Message - From: Tolik Litovsky tlito...@redhat.com To: ee...@redhat.com Cc: infra@ovirt.org Sent: Thursday, June 4, 2015 9:28:27 AM Subject: RE: Access to ovirt-srv08.ovirt.org The question is the nesting depth. We run currently in depth 2 . VM inside VM. Will depth of 3 will work. VM slave, and another 2 VM nesting above it. KVM people? Plz help. adding devel, but please elaborate on the depth, does it have to be nested or you're doing it because you only had 1 bare metal host till now (think you'll have 2 nested vms for example) The test requires depth 2 nested . We create an engine with hosts and storage. And on top of them we run another VM to test the ovirt features. This is the test scenario. OK, then we need to run a test and see the performance for it, because i think we will we gain many benefits from it, in terms of management and stability. Eyal , this testenv will most likely not be in my responsibility longer. But I know that there are several bare metals dedicated to this jobs. Can we plz bring them up , I belive it will be very helpfull before we finish the nested slaves investigation. There's only one afaik, and you are already using it. there are 6 more servers, but they are not ready yet, we will start working on them soon, i hope they will be ready in a week's time, we will update. regardless the system test maintainer should run the nested test so we'll know if we can leverage this. I was not aware they were dedicated for those jobs some of them were ordered to help with this of course, and since there will be a very big initiative of writing new tests soon, i assume one host won't be enough, we will allocate 2 more hosts for now, and will review the load and decide on what to do next. Tolik. Original message From: Eyal Edri ee...@redhat.com Date: 03/06/2015 11:14 PM (GMT+02:00) To: tlito...@redhat.com Cc: infra@ovirt.org Subject: Re: Access to ovirt-srv08.ovirt.org I'd like to move the system testing framework to use nested vms instead of directly using the baremetal hosts. we did use a single host to POC the system, i think now that we're about to expand it will be much more managble and scalable if we'll start using nested VMs which will be managed via the hosted engine. e. - Original Message - From: Tolik Litvosky tlito...@redhat.com To: infra@ovirt.org Sent: Wednesday, June 3, 2015 3:59:36 PM Subject: Access to ovirt-srv08.ovirt.org Hi I need access to the ovirt-srv08.ovirt.org in order to update the testenv CI repo directory. -- Best regards Tolik Litovsky RHEV-H Team Red Hat Red Hat: trustworthy, transformative technology. Powered by the community. Connect at redhat.com ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra -- Best regards Tolik Litovsky RHEV-H Team Red Hat Red Hat: trustworthy, transformative technology. Powered by the community. Connect at redhat.com -- Best regards Tolik Litovsky RHEV-H Team Red Hat Red Hat: trustworthy, transformative technology.
Re: [ovirt-devel] gerrit+ci improvement proposal
- Original Message - From: Oved Ourfali ov...@redhat.com To: Eyal Edri ee...@redhat.com Cc: de...@ovirt.org, infra@ovirt.org Sent: Thursday, June 4, 2015 10:03:02 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal - Original Message - From: Eyal Edri ee...@redhat.com To: Sandro Bonazzola sbona...@redhat.com Cc: infra@ovirt.org, de...@ovirt.org Sent: Thursday, June 4, 2015 9:46:40 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal - Original Message - From: Sandro Bonazzola sbona...@redhat.com To: Eyal Edri ee...@redhat.com, Max Kovgan mkov...@redhat.com Cc: de...@ovirt.org, infra@ovirt.org Sent: Thursday, June 4, 2015 9:11:10 AM Subject: Re: [ovirt-devel] gerrit+ci improvement proposal Il 03/06/2015 21:46, Eyal Edri ha scritto: - Original Message - From: Max Kovgan mkov...@redhat.com To: de...@ovirt.org Cc: infra@ovirt.org Sent: Wednesday, June 3, 2015 8:22:54 PM Subject: [ovirt-devel] gerrit+ci improvement proposal Hi everyone! We really want to have reliable and snappy CI: to allow short cycles and encourage developers to write tests. # Problem Many patches are neither ready for review nor for CI upon submission, which is OK. But running all the jobs on those patches with limited resources results in: overloaded resources, slow response time, unhappy developers. # Proposed Solution To run less jobs we know we don’t need to, thus making more resources for the jobs we need to run. We have been experimenting to make our CI stabler and quicker to respond by using gerrit flags. This has improved in both directions very well internally. Now it seems a good time to let all the oVirt projects to use this. This solution indirectly promotes reviews and quick tests - “to fail early”, yet full blown static code analysis and long tests to run “when ready”. # How it works 2 new gerrit independent flags are added to gerrit. ## CI flag Will express patch CI status. Values: * +1 CI passed * 0 CI did not run yet * -1 CI failed Permissions for setting: project maintainers (for special cases) should be able to set/override (except Jenkins). ## Workflow flag Will express patch “workflow” state. Values: * 0 Work In Progress * +1 Ready For Review * +2 Ready For Merge Permissions for setting: Owner can set +1, Project Maintainers can set +2 ## Review + CI Integration: Merging [“Submit” button to appear] will require: Review+1, CI+1, Workflow+2 Patch lifecycle now is: --- patch state |owner |reviewer |maintainer |CI tests |pass --- added/updated |- |-|- |quick|CI+1 review|Workflow+1|Review+1 |- |heavy |CI+1 merge ready |- |-|Workflow+2 |gating |CI+1 merge |- |-|merge |merge|CI+1 Changes from current workflow: Owner only adds reviewers, now owner needs to set Workflow+1 for the patch to be reviewed, and heavily auto-tested. Maintainer now needs to set Workflow+2 and wait for Submit button to appear after CI has completed running gating tests. Next step will be to automate merge the change after Workflow+2 has been set by the Maintainer and gating tests passed. ## Why now? It is elimination of waste. The sooner - the better. The solution has been used for a while and it works. Resolving the problem without gerrit involved will lead to adding unreliable code into jobs, and will still be prone to problems: Just recently, 3d ago we’ve tried detecting what to run from jenkins relying only on gerrit comments so that upon Verified+1, we’d run the job. We could not use “Review+1”, because it makes no sense at all, so we left the job to set Verified+1. Meaning - re-trigger itself immediately more than 1 times. Jenkins and its visitors very unhappy, and we had to stop those jobs, clean up the queue, and spam developers. ## OK OK OK. Now what? Now we want your comments and opinions before pushing this further: Please participate in this thread, so we can start trying it out. Ask, Suggest better ideas, all this is welcome. Best Regards! N.B. Of course, this is not written in stone, in case we find a better approach on solving those issues, we will change to it. And we will keep improving so don't be afraid that it will be enforced: if this does not work
