Jira (PUP-7014) Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 20000
Title: Message Title Michael Smith commented on PUP-7014 Re: Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 2 We have not yet enabled running actual tests with MCO. Most of the work preparing that is merged, however. I think it needs a few tweaks to ci-job-configs (QA-2496). Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7014) Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 20000
Title: Message Title Ethan Brown commented on PUP-7014 Re: Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 2 1. Ok, so given current information, we don't know why this happened. That's unfortunate and I think our only response can be to exercise more vigilance when looking at green pipelines. 2. Great! 3. ci:test:quick was added as a part of https://tickets.puppetlabs.com/browse/PUP-5105 - it shouldn't be considered for this ticket though given it's used for acceptance tests. This ticket is about specs. 4. The MCO issue I swear we've seen before - Josh Cooper, Michael Smith or John Duarte might know more about that one. I thought that we had removed the echo true in favor of executing the few tests in the suite. But I might be mistaken there? Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7014) Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 20000
Title: Message Title Glenn Sarti commented on PUP-7014 Re: Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 2 1. I can't RCA the issue of only running the 1900 tests as the logs for that job have been lost and it has not occurred since. 2. I've updated the Hiera and MCO job configs so they now output Test Results into Jenkins so at least the test count can be audited 3. The ci:test:quick behaviour is not documented anywhere that I could find. I trawled through the source code to find that. Note that the full set of acceptance tests is run, but as part of the puppet-agent acceptance suite, not part of puppet. This looks like it's just a smoke acceptance test run. I cannot see this quick code in any of our other pipelines 4. I have noticed that the MCO acceptance tests (https://jenkins-master-prod-1.delivery.puppetlabs.net/job/platform_marionette-collective_intn-van-sys_master/configure) doesn't actually run anything. echo "" echo "Running Tests:" echo "true" echo "" Add Comment
Jira (PUP-7014) Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 20000
Title: Message Title Ethan Brown commented on PUP-7014 Re: Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 2 Glenn Sarti is your final assessment that the cause of this was due to only running against "recently modified tests"? As we briefly discussed, I'm not sure that that approach is a good heuristic for determining which tests to run. Do we have the variable nature documented somewhere? How many pipelines does it impact? etc etc Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7014) Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 20000
Title: Message Title Glenn Sarti assigned an issue to Ethan Brown Puppet / PUP-7014 Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 2 Change By: Glenn Sarti Assignee: Ethan Brown Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7014) Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 20000
Title: Message Title Glenn Sarti commented on PUP-7014 Re: Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 2 Ethan Brown Is there anything else you wanted me to look at? Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7014) Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 20000
Title: Message Title Glenn Sarti commented on PUP-7014 Re: Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 2 Raised maint PR https://github.com/puppetlabs/ci-job-configs/pull/2170 so that MCO and Hiera emit spec unit test results. Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7014) Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 20000
Title: Message Title Glenn Sarti commented on PUP-7014 Re: Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 2 The puppet acceptance test don't get a full run until Step 7 of the puppet-agent pipeline. Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7014) Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 20000
Title: Message Title Glenn Sarti commented on PUP-7014 Re: Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 2 The puppet acceptance jobs run the ci:test:quick rake task which runs a dynamic set of tests. So my method of auditing was incorrect. 15 tests plus any tests that have been modified in the last 2 weeks https://github.com/puppetlabs/puppet/blob/master/acceptance/Rakefile#L277-L310 Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7014) Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 20000
Title: Message Title Glenn Sarti commented on PUP-7014 Re: Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 2 Actions The latest run of the puppet acceptance tests mysteriously didn't run 9 tests. Will investigate The following jobs do not output job XML for Jenkins to consume so we're missing out on build statistics etc. on them platform_marionette-collective_unit-ruby_*** platform_marionette-collective_intn-van-sys_*** platform_hiera_unit-ruby_*** Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7014) Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 20000
Title: Message Title Glenn Sarti commented on PUP-7014 Re: Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 2 I wrote some powershell to extract all of the test results from Jenkins and then process them to look for anomalies. The scripts are attached. To detect anomalies I just took the average of the Total number of tests for a job, and then checked that each single build was +/- 5% of that average. Results are below: Average tests for job platform_facter_intn-van-sys_master is 59 *** Job platform_hiera_unit-ruby_master has no job stats! Average tests for job platform_hiera_intn-van-sys_master is 9 *** Job platform_marionette-collective_unit-ruby_master has no job stats! *** Job platform_marionette-collective_intn-van-sys_master has no job stats! Average tests for job platform_puppet_unit-ruby-win_master is 21889.5 Average tests for job platform_puppet_unit-ruby_master is 22357.2619047619 Average tests for job platform_puppet_intn-van-sys_master is 32.8461538461538
Jira (PUP-7014) Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 20000
Title: Message Title Glenn Sarti updated an issue Puppet / PUP-7014 Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 2 Change By: Glenn Sarti Attachment: ProcessCSV.ps1 Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7014) Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 20000
Title: Message Title Glenn Sarti updated an issue Puppet / PUP-7014 Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 2 Change By: Glenn Sarti Attachment: AuditPuppetAgentTests.ps1 Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7014) Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 20000
Title: Message Title Glenn Sarti assigned an issue to Glenn Sarti Puppet / PUP-7014 Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 2 Change By: Glenn Sarti Assignee: Glenn Sarti Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7014) Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 20000
Title: Message Title Glenn Sarti commented on PUP-7014 Re: Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 2 Unfortunately Build 52 has been purged so an RCA is going to be very difficult. Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7014) Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 20000
Title: Message Title Kurt Wall updated an issue Puppet / PUP-7014 Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 2 Change By: Kurt Wall QA Risk Assessment Reason: [Bulk update] Tickets are for build, infra, or docs and so not risk assessed QA Risk Assessment: Needs Assessment No Action Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7014) Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 20000
Title: Message Title Ethan Brown updated an issue Puppet / PUP-7014 Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 2 Change By: Ethan Brown It was determined that the new CI.next pipelines were not always running the full spec suite on Windows - at job https://jenkins-master-prod-1.delivery.puppetlabs.net/job/platform_puppet_unit-ruby-win_master/Up through [job 52|https://jenkins-master-prod-1.delivery.puppetlabs.net/job/platform_puppet_unit-ruby-win_master/52/] on 12/12/2016, we were getting false positive passing runs because only about 1900 tests were running out of the 20,000+ tests.The pipeline started failing as of [job 53](https://jenkins-master-prod-1.delivery.puppetlabs.net/job/platform_puppet_unit-ruby-win_master/53/) when the full spec suite started running properly (the actual test bugs were fixed in https://github.com/puppetlabs/puppet/pull/5431).It is currently unknown why the full suite has not been running, and this ticket is for:* determining why the suites ran a reduced set of specs (its possible this is related to Cygwin git being used on these new CI spec boxes? Note that ticket https://tickets.puppetlabs.com/browse/IMAGES-431 exists for updaiting vmpooler / acceptance nodes to not use Cygwin git, but doesn't address spec boxes)* are Windows spec jobs the only affected ones* should other matrices in CI.next be audited to confirm they have been migrated properly?/cc [~josh] [~samuel] Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You
Jira (PUP-7014) Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 20000
Title: Message Title Ethan Brown updated an issue Puppet / PUP-7014 Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 2 Change By: Ethan Brown It was determined that the new CI.next pipelines were not always running the full spec suite on Windows - at job https://jenkins-master-prod-1.delivery.puppetlabs.net/job/platform_puppet_unit-ruby-win_master/Up through [job 52|https://jenkins-master-prod-1.delivery.puppetlabs.net/job/platform_puppet_unit-ruby-win_master/52/] on 12/12/2016, we were getting false positive passing runs because only about 1900 tests were running out of the 20,000+ tests.The pipeline started failing as of [job 53 ]( | https://jenkins-master-prod-1.delivery.puppetlabs.net/job/platform_puppet_unit-ruby-win_master/53/ ) ] when the full spec suite started running properly (the actual test bugs were fixed in https://github.com/puppetlabs/puppet/pull/5431).It is currently unknown why the full suite has not been running, and this ticket is for:* determining why the suites ran a reduced set of specs (its possible this is related to Cygwin git being used on these new CI spec boxes? Note that ticket https://tickets.puppetlabs.com/browse/IMAGES-431 exists for updaiting vmpooler / acceptance nodes to not use Cygwin git, but doesn't address spec boxes)* are Windows spec jobs the only affected ones* should other matrices in CI.next be audited to confirm they have been migrated properly?/cc [~josh] [~samuel] Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You
Jira (PUP-7014) Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 20000
Title: Message Title Ethan Brown updated an issue Puppet / PUP-7014 Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 2 Change By: Ethan Brown Labels: CI Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7014) Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 20000
Title: Message Title Moses Mendoza updated an issue Puppet / PUP-7014 Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 2 Change By: Moses Mendoza Team: Agent & Platform Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7014) Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 20000
Title: Message Title Ethan Brown created an issue Puppet / PUP-7014 Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 2 Issue Type: Task Assignee: Unassigned Created: 2016/12/14 11:00 AM Priority: Normal Reporter: Ethan Brown Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at
