Re: [openstack-dev] [Nova] Turbo hipster problems

[Gary Kotton]

Thanks. Yes, it is up and running again!

From: Joshua Hesketh 
mailto:joshua.hesk...@rackspace.com>>
Reply-To: OpenStack List 
mailto:openstack-dev@lists.openstack.org>>
Date: Monday, October 20, 2014 at 3:45 AM
To: OpenStack List 
mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [Nova] Turbo hipster problems

Hi Gary,

Sorry we had a mis-hap over the weekend. The Database CI should be back up and 
running now. Let me know if you see any more problems.

Cheers,
Josh

Rackspace Australia

On 10/18/14 1:55 AM, Gary Kotton wrote:
Hi,
Anyone aware why Turbo his peter is failing with:

real-db-upgrade_nova_percona_user_002:th-percona<https://urldefense.proofpoint.com/v1/url?u=http://localhost&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZFxPEq8%3D%0A&m=HSN3f4I4FiDfec3IQQMovHsZInTYKJB%2BnFTFljbiNsg%3D%0A&s=63ef5a3f5bd643d5565a9fa0b8df3c47dceafc269f5682f6b63a92ebd076902d>Exception:
 [Errno 2] No such file or directory: 
'/var/lib/turbo-hipster/datasets_user_002' in 0s

Thanks
Gary



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Turbo hipster problems

[Joshua Hesketh]

Hi Gary,

Sorry we had a mis-hap over the weekend. The Database CI should be back 
up and running now. Let me know if you see any more problems.


Cheers,
Josh

Rackspace Australia

On 10/18/14 1:55 AM, Gary Kotton wrote:

Hi,
Anyone aware why Turbo his peter is failing with:

real-db-upgrade_nova_percona_user_002:th-percona  
Exception: [Errno 2] No such file or directory: 
'/var/lib/turbo-hipster/datasets_user_002' in 0s


Thanks
Gary


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Turbo hipster

[Gary Kotton]

Thanks!

From: Joshua Hesketh 
mailto:joshua.hesk...@rackspace.com>>
Reply-To: OpenStack List 
mailto:openstack-dev@lists.openstack.org>>
Date: Sunday, June 29, 2014 at 4:25 PM
To: OpenStack List 
mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [Nova] Turbo hipster

Hi Gary,

Thanks for the notification. Our nodepool had built a corrupt image which was 
returning false positives. I have fixed this up and rerun tests on changes that 
had negative votes where Jenkins passed. If you notice any changes that seem 
like a false negative please issue a "recheck migrations". If it fails a second 
time please let us know at rc...@rcbops.com<mailto:rc...@rcbops.com>.

FYI, you can see our testing progress at 
http://zuul.rcbops.com<https://urldefense.proofpoint.com/v1/url?u=http://zuul.rcbops.com&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZFxPEq8%3D%0A&m=RJuFlqY3B%2Feoge51zD7bpiFA3kV56%2BHZ%2FbroCaFAdTk%3D%0A&s=4beefa50e4c44dc8ebd748c9198475935b4104c4080e6e41bcb969bd84bf6175>

Cheers,
Josh

Rackspace Australia

On 6/29/14 5:48 PM, Gary Kotton wrote:
Hi,
This appears to be broken over the last 24 hours.
Thanks
Gary



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Turbo hipster

[Joshua Hesketh]

Hi Gary,

Thanks for the notification. Our nodepool had built a corrupt image 
which was returning false positives. I have fixed this up and rerun 
tests on changes that had negative votes where Jenkins passed. If you 
notice any changes that seem like a false negative please issue a 
"recheck migrations". If it fails a second time please let us know at 
rc...@rcbops.com.


FYI, you can see our testing progress at http://zuul.rcbops.com

Cheers,
Josh

Rackspace Australia

On 6/29/14 5:48 PM, Gary Kotton wrote:

Hi,
This appears to be broken over the last 24 hours.
Thanks
Gary


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][turbo hipster] unable to rebase

[Michael Still]

Hi!

Sorry the slow response. I've been at a conference this week, so I'm a
bit behind. I apologise for that.

The root cause here is a failure to understand the relatively
undocumented git behaviour of zuul. We think we've now got a fix, and
will deploy it soon.

Michael


On Thu, Jan 9, 2014 at 8:30 PM, John Garbutt  wrote:
> On 8 January 2014 12:07, Gary Kotton  wrote:
>> Hi,
>> Patch sets that are cascaded has have the following errors:
>>
>> "Database migration testing failed either due to migrations unable to be
>> applied correctly or taking too long.
>>
>> This change was unable to be automatically merged with the current state of
>> the repository. Please rebase your change and upload a new patch set."
>>
>> What do you suggest?
>
> If you rebase you code, would that not fix everything?
>
> It seems for the tests to pass, that change needs to be able to rebase
> on trunk these days.
>
> John
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Rackspace Australia

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][turbo hipster] unable to rebase

[John Garbutt]

On 8 January 2014 12:07, Gary Kotton  wrote:
> Hi,
> Patch sets that are cascaded has have the following errors:
>
> "Database migration testing failed either due to migrations unable to be
> applied correctly or taking too long.
>
> This change was unable to be automatically merged with the current state of
> the repository. Please rebase your change and upload a new patch set."
>
> What do you suggest?

If you rebase you code, would that not fix everything?

It seems for the tests to pass, that change needs to be able to rebase
on trunk these days.

John

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Turbo-hipster

[James E. Blair]

Dan Prince  writes:

> - Original Message -
>> From: "Michael Still" 
>> - commenting "recheck .*"
>> - commenting "recheck migrations"
>
> With the growing interest in 3rd party testing systems would using 'recheck 
> turbo-hipster' make more sense here?
>
> I'm fine with 'recheck migrations' in addition for turbo-hipster but it would 
> make sense to align the recheck naming scheme with the title of the reviewer 
> for the 3rd party testing system.

This is the can of worms I was hoping we would not open.  Or try to get
them all back into the can and close it again is perhaps the better
metaphor.

I do not think that system-specific recheck commands will be actually
that useful or important.  As I mentioned, I understand the theoretical
usefulness of being able to say "oops, I can tell this one system messed
up, let's ask it to try again".  But given that case is covered by
asking all systems to recheck, it seems harmless to say "all systems to
recheck".

I just don't think that asking developers to learn a micro-language of
commands left in gerrit comments is the best use of time.  Maybe I'm
wrong about that, but I was hoping we could try not creating it first to
see if there's really a need.  Dealing with the expected errors of a
system first coming online doesn't, in my mind, demonstrate that need.

If it turns out that asking programmers not to create a new language is
futile, yes, I'd rather we have some predictability.  Most third party
CI systems have descriptive names, so rather than saying "recheck
turbo-hipster", perhaps we should change turbo-hipster's display name to
something related to "migrations".

So _if_ we want to have system-specific recheck commands, then I think
we should ask operators to make them in the form "recheck ",
and that they output a brief sentence of help text to remind people of
that in the report message in Gerrit.

-Jim

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Turbo-hipster

[Dan Prince]

- Original Message -
> From: "Michael Still" 
> To: "OpenStack Development Mailing List (not for usage questions)" 
> 
> Sent: Thursday, January 2, 2014 7:48:16 PM
> Subject: Re: [openstack-dev] [nova] Turbo-hipster
> 
> On Fri, Jan 3, 2014 at 9:39 AM, Michael Still  wrote:
> > On Fri, Jan 3, 2014 at 9:24 AM, James E. Blair 
> > wrote:
> >
> >> However, there are _a lot_ of third-party test systems coming on-line,
> >> and I'm not sure that expanding the "recheck language" to support ever
> >> more complexity is a good idea.  I can see how being able to say
> >> "recheck foo" would be useful in some circumstances, but given that just
> >> saying "recheck" will suffice, I'd prefer that we kept the general
> >> recommendation simple so developers can worry about something else.
> >
> > Fair enough. I feel like you and I should sit down and chat about this
> > stuff at the LCA meetup next week. If we need to make tweaks based on
> > that, then we will.
> 
> Further to this, I have just reloaded our zuul with a rules change.
> The following events will all cause a turbo hipster check run:
> 
> - uploading a patchset
> - restoring a patchset
> - commenting "recheck .*"
> - commenting "recheck migrations"

With the growing interest in 3rd party testing systems would using 'recheck 
turbo-hipster' make more sense here?

I'm fine with 'recheck migrations' in addition for turbo-hipster but it would 
make sense to align the recheck naming scheme with the title of the reviewer 
for the 3rd party testing system.


> 
> Cheers,
> Michael
> 
> --
> Rackspace Australia
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Turbo-hipster

[Michael Still]

On Fri, Jan 3, 2014 at 9:39 AM, Michael Still  wrote:
> On Fri, Jan 3, 2014 at 9:24 AM, James E. Blair  wrote:
>
>> However, there are _a lot_ of third-party test systems coming on-line,
>> and I'm not sure that expanding the "recheck language" to support ever
>> more complexity is a good idea.  I can see how being able to say
>> "recheck foo" would be useful in some circumstances, but given that just
>> saying "recheck" will suffice, I'd prefer that we kept the general
>> recommendation simple so developers can worry about something else.
>
> Fair enough. I feel like you and I should sit down and chat about this
> stuff at the LCA meetup next week. If we need to make tweaks based on
> that, then we will.

Further to this, I have just reloaded our zuul with a rules change.
The following events will all cause a turbo hipster check run:

- uploading a patchset
- restoring a patchset
- commenting "recheck .*"
- commenting "recheck migrations"

Cheers,
Michael

-- 
Rackspace Australia

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Turbo-hipster

[Anita Kuno]

On 01/02/2014 05:58 PM, Michael Still wrote:
> On Fri, Jan 3, 2014 at 9:46 AM, Anita Kuno  wrote:
> 
>> I'd love to attend this chat, if possible. A number of the coming
>> on-line third-party test systems are motivated by neutron plugins. I'd
>> get a lot from hearing this discussion.
> 
> I've created a BoF on Monday night straight after the CI miniconf in
> the same room. Let's all get together and have a chat and then go to
> dinner.
> 
> Michael
> 
Great, thank you Michael.

Anita.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Turbo-hipster

[Michael Still]

On Fri, Jan 3, 2014 at 9:46 AM, Anita Kuno  wrote:

> I'd love to attend this chat, if possible. A number of the coming
> on-line third-party test systems are motivated by neutron plugins. I'd
> get a lot from hearing this discussion.

I've created a BoF on Monday night straight after the CI miniconf in
the same room. Let's all get together and have a chat and then go to
dinner.

Michael

-- 
Rackspace Australia

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Turbo-hipster

[Anita Kuno]

On 01/02/2014 05:39 PM, Michael Still wrote:
> On Fri, Jan 3, 2014 at 9:24 AM, James E. Blair  wrote:
> 
>> However, there are _a lot_ of third-party test systems coming on-line,
>> and I'm not sure that expanding the "recheck language" to support ever
>> more complexity is a good idea.  I can see how being able to say
>> "recheck foo" would be useful in some circumstances, but given that just
>> saying "recheck" will suffice, I'd prefer that we kept the general
>> recommendation simple so developers can worry about something else.
> 
> Fair enough. I feel like you and I should sit down and chat about this
> stuff at the LCA meetup next week. If we need to make tweaks based on
> that, then we will.
> 
> Cheers,
> Michael
> 
I'd love to attend this chat, if possible. A number of the coming
on-line third-party test systems are motivated by neutron plugins. I'd
get a lot from hearing this discussion.

Thanks,
Anita.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Turbo-hipster

[Michael Still]

On Fri, Jan 3, 2014 at 9:24 AM, James E. Blair  wrote:

> However, there are _a lot_ of third-party test systems coming on-line,
> and I'm not sure that expanding the "recheck language" to support ever
> more complexity is a good idea.  I can see how being able to say
> "recheck foo" would be useful in some circumstances, but given that just
> saying "recheck" will suffice, I'd prefer that we kept the general
> recommendation simple so developers can worry about something else.

Fair enough. I feel like you and I should sit down and chat about this
stuff at the LCA meetup next week. If we need to make tweaks based on
that, then we will.

Cheers,
Michael

-- 
Rackspace Australia

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Turbo-hipster

[Robert Collins]

On 3 January 2014 11:26, James E. Blair  wrote:

> If you are able to do this and benchmark the performance of a cloud
> server reliably enough, we might be able to make progress on performance
> testing, which has been long desired.  The large ops test is (somewhat
> accidentally) a performance test, and predictably, it has failed when we
> change cloud node provider configurations.  A benchmark could make this
> test more reliable and other tests more feasible.

In bzr we found it much more reliable to do tests that isolate and
capture the *effort*, not the time: most [not all] performance issues
have both a time and effort domain, and the effort domain is usually
correlated with time in a particular environment, but itself
approximately constant across environments.

For instance, MB sent in a request, or messages on the message bus, or
writes to the file system, or queries sent to the DB.

So the structure we ended up with - which was quite successful - was:
 - a cron job based benchmark that ran several versions through
functional scenarios and reporting timing data
 - gating tests that tested effort for operations
 - a human process whereby someone wanting to put a ratchet on some
aspect of performance would write an effort based test or three to
capture the status quo, then make it better and update the tests with
their improvements.

I think this would work well for OpenStack too - and infact we have
some things that are in this general direction already.

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Turbo-hipster

[James E. Blair]

Sean Dague  writes:

> On 01/02/2014 04:29 PM, Michael Still wrote:
>> Heh, I didn't know that wiki page existed. I've added an entry to the 
>> checklist.
>> 
>> There's also some talk of adding some help text to the vote message
>> turbo-hipster leaves in gerrit, but we haven't gotten around to doing
>> that yet.
>> 
>> Cheers,
>> Michael
>
> So was there enough countable slowness earlier in the run that you could
> have predicted these runs would be slower overall?
>
> My experience looking at Tempest run data is there can be as much as an
> +60% variance from fastest and slowest nodes (same instance type) within
> the same cloud provider, which is the reason we've never tried to
> performance gate on it.
>
> However if there was some earlier benchmark that would let you realize
> that the whole run was slow, so give it more of a buffer, that would
> probably be useful.

If you are able to do this and benchmark the performance of a cloud
server reliably enough, we might be able to make progress on performance
testing, which has been long desired.  The large ops test is (somewhat
accidentally) a performance test, and predictably, it has failed when we
change cloud node provider configurations.  A benchmark could make this
test more reliable and other tests more feasible.

-Jim

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Turbo-hipster

[James E. Blair]

Michael Still  writes:

> Heh, I didn't know that wiki page existed. I've added an entry to the 
> checklist.
>
> There's also some talk of adding some help text to the vote message
> turbo-hipster leaves in gerrit, but we haven't gotten around to doing
> that yet.

I would rather not mention it on that page, which is the documentation
for the project gating system and developer workflow (Zuul links to it
when it leaves a failure message) so I have removed it.

I _do_ think adding help text to the messages third-party tools leave,
and/or linking to specific documentation (ideally also in the OpenStack
wiki) from there is a good idea.

However, there are _a lot_ of third-party test systems coming on-line,
and I'm not sure that expanding the "recheck language" to support ever
more complexity is a good idea.  I can see how being able to say
"recheck foo" would be useful in some circumstances, but given that just
saying "recheck" will suffice, I'd prefer that we kept the general
recommendation simple so developers can worry about something else.

Certainly at a minimum, "recheck" should recheck all the systems; that's
one of the proposed requirements here:

  https://review.openstack.org/#/c/63478/5/doc/source/third_party.rst

I think it would be best if we stopped there.  But if you still feel
very strongly that you want a private extension to the syntax, please
consider how necessary it is for most developers to know about it when
you decide how prominently to feature it in messages or documentation
about your tools.

-Jim

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Turbo-hipster

[Sean Dague]

On 01/02/2014 04:29 PM, Michael Still wrote:
> Heh, I didn't know that wiki page existed. I've added an entry to the 
> checklist.
> 
> There's also some talk of adding some help text to the vote message
> turbo-hipster leaves in gerrit, but we haven't gotten around to doing
> that yet.
> 
> Cheers,
> Michael

So was there enough countable slowness earlier in the run that you could
have predicted these runs would be slower overall?

My experience looking at Tempest run data is there can be as much as an
+60% variance from fastest and slowest nodes (same instance type) within
the same cloud provider, which is the reason we've never tried to
performance gate on it.

However if there was some earlier benchmark that would let you realize
that the whole run was slow, so give it more of a buffer, that would
probably be useful.

-Sean

-- 
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Turbo-hipster

[Michael Still]

Heh, I didn't know that wiki page existed. I've added an entry to the checklist.

There's also some talk of adding some help text to the vote message
turbo-hipster leaves in gerrit, but we haven't gotten around to doing
that yet.

Cheers,
Michael

On Fri, Jan 3, 2014 at 12:57 AM, Matt Riedemann
 wrote:
>
>
> On 12/31/2013 3:58 PM, Michael Still wrote:
>>
>> Hi.
>>
>> So while turbo hipster is new, I've been reading every failure message
>> it produces to make sure its not too badly wrong. There were four
>> failures posted last night while I slept:
>>
>> https://review.openstack.org/#/c/64521
>> 
>>
>> This one is a TH bug. We shouldn't be testing stable branches.
>> bug/1265238 has been filed to track this.
>>
>> https://review.openstack.org/#/c/61753
>> 
>>
>> This is your review. The failed run's log is
>>
>> https://ssl.rcbops.com/turbo_hipster/logviewer/?q=/turbo_hipster/results/61/61753/8/check/gate-real-db-upgrade_nova_percona_user_001/1326092/user_001.log
>> and you can see from the failure message that migrations 152 and 206
>> took "too long".
>>
>> Migration 152 took 326 seconds, where our historical data of 2,846
>> test migrations says it should take 222 seconds. Migration 206 took 81
>> seconds, where we think it should take 56 seconds based on 2,940 test
>> runs.
>>
>> Whilst I can't explain why those migrations took too long this time
>> around, they are certainly exactly the sort of thing TH is meant to
>> catch. If you think your patch isn't responsible (perhaps the machine
>> is just being slow or something), you can always retest by leaving a
>> review comment of "recheck migrations". I have done this for you on
>> this patch.
>
>
> Michael, is "recheck migrations" something that is going to be added to the
> wiki for test failures here?
>
> https://wiki.openstack.org/wiki/GerritJenkinsGit#Test_Failures
>
>
>>
>> https://review.openstack.org/#/c/61717
>> 
>>
>> This review also had similar unexplained slowness, but has already
>> been rechecked by someone else and now passes. I note that the
>> slowness in both cases was from the same TH worker node, and I will
>> keep an eye on that node today.
>>
>> https://review.openstack.org/#/c/56420
>> 
>>
>> This review also had slowness in migration 206, but has been rechecked
>> by the developer and now passes. It wasn't on the percona-001 worker
>> that the other two were on, so perhaps this indicates that we need to
>> relax the timing requirements for migration 206.
>>
>> Hope this helps,
>> Michael
>>
>> On Wed, Jan 1, 2014 at 12:34 AM, Gary Kotton  wrote:
>>>
>>> Hi,
>>> It seems that she/he is behaving oddly again. I have posted a patch that
>>> does not have any database changes and it has give me a –1….
>>> Happy new year
>>> Gary
>>>
>>> ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>>
>
> --
>
> Thanks,
>
> Matt Riedemann
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Rackspace Australia

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Turbo-hipster

[Matt Riedemann]

On 12/31/2013 3:58 PM, Michael Still wrote:

Hi.

So while turbo hipster is new, I've been reading every failure message
it produces to make sure its not too badly wrong. There were four
failures posted last night while I slept:

https://review.openstack.org/#/c/64521


This one is a TH bug. We shouldn't be testing stable branches.
bug/1265238 has been filed to track this.

https://review.openstack.org/#/c/61753


This is your review. The failed run's log is
https://ssl.rcbops.com/turbo_hipster/logviewer/?q=/turbo_hipster/results/61/61753/8/check/gate-real-db-upgrade_nova_percona_user_001/1326092/user_001.log
and you can see from the failure message that migrations 152 and 206
took "too long".

Migration 152 took 326 seconds, where our historical data of 2,846
test migrations says it should take 222 seconds. Migration 206 took 81
seconds, where we think it should take 56 seconds based on 2,940 test
runs.

Whilst I can't explain why those migrations took too long this time
around, they are certainly exactly the sort of thing TH is meant to
catch. If you think your patch isn't responsible (perhaps the machine
is just being slow or something), you can always retest by leaving a
review comment of "recheck migrations". I have done this for you on
this patch.


Michael, is "recheck migrations" something that is going to be added to 
the wiki for test failures here?


https://wiki.openstack.org/wiki/GerritJenkinsGit#Test_Failures



https://review.openstack.org/#/c/61717


This review also had similar unexplained slowness, but has already
been rechecked by someone else and now passes. I note that the
slowness in both cases was from the same TH worker node, and I will
keep an eye on that node today.

https://review.openstack.org/#/c/56420


This review also had slowness in migration 206, but has been rechecked
by the developer and now passes. It wasn't on the percona-001 worker
that the other two were on, so perhaps this indicates that we need to
relax the timing requirements for migration 206.

Hope this helps,
Michael

On Wed, Jan 1, 2014 at 12:34 AM, Gary Kotton  wrote:

Hi,
It seems that she/he is behaving oddly again. I have posted a patch that
does not have any database changes and it has give me a –1….
Happy new year
Gary

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev







--

Thanks,

Matt Riedemann


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Turbo-hipster

[Gary Kotton]

Thanks for the detailed update. This is very informative and useful.
Thanks

On 12/31/13 11:58 PM, "Michael Still"  wrote:

>Hi.
>
>So while turbo hipster is new, I've been reading every failure message
>it produces to make sure its not too badly wrong. There were four
>failures posted last night while I slept:
>
>https://urldefense.proofpoint.com/v1/url?u=https://review.openstack.org/%2
>3/c/64521&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=eH0pxTUZo8NPZyF6hgoMQu%2Bf
>Dtysg45MkPhCZFxPEq8%3D%0A&m=QjlZEwdZOsbG6HW%2FAwjBP1GxhEprFcAcOx%2Btg9BHc4
>Q%3D%0A&s=07845bb3c782af5c62d23c9234df0023cd3d37798ee2ae06d96ac2691419a647
>
>
>This one is a TH bug. We shouldn't be testing stable branches.
>bug/1265238 has been filed to track this.
>
>https://urldefense.proofpoint.com/v1/url?u=https://review.openstack.org/%2
>3/c/61753&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=eH0pxTUZo8NPZyF6hgoMQu%2Bf
>Dtysg45MkPhCZFxPEq8%3D%0A&m=QjlZEwdZOsbG6HW%2FAwjBP1GxhEprFcAcOx%2Btg9BHc4
>Q%3D%0A&s=c332d74b6bf0092724c0d10227d71f5ac8df6d0e48ca53d48054503c2e03d031
>
>
>This is your review. The failed run's log is
>https://urldefense.proofpoint.com/v1/url?u=https://ssl.rcbops.com/turbo_hi
>pster/logviewer/?q%3D/turbo_hipster/results/61/61753/8/check/gate-real-db-
>upgrade_nova_percona_user_001/1326092/user_001.log&k=oIvRg1%2BdGAgOoM1BIlL
>Lqw%3D%3D%0A&r=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZFxPEq8%3D%0A&m=QjlZE
>wdZOsbG6HW%2FAwjBP1GxhEprFcAcOx%2Btg9BHc4Q%3D%0A&s=de964203fc8fb25cd20a54a
>ae34e35a175c3e9bb325e0d442b9a7b7ced78a8ad
>and you can see from the failure message that migrations 152 and 206
>took "too long".
>
>Migration 152 took 326 seconds, where our historical data of 2,846
>test migrations says it should take 222 seconds. Migration 206 took 81
>seconds, where we think it should take 56 seconds based on 2,940 test
>runs.
>
>Whilst I can't explain why those migrations took too long this time
>around, they are certainly exactly the sort of thing TH is meant to
>catch. If you think your patch isn't responsible (perhaps the machine
>is just being slow or something), you can always retest by leaving a
>review comment of "recheck migrations". I have done this for you on
>this patch.
>
>https://urldefense.proofpoint.com/v1/url?u=https://review.openstack.org/%2
>3/c/61717&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=eH0pxTUZo8NPZyF6hgoMQu%2Bf
>Dtysg45MkPhCZFxPEq8%3D%0A&m=QjlZEwdZOsbG6HW%2FAwjBP1GxhEprFcAcOx%2Btg9BHc4
>Q%3D%0A&s=a4bc17a0cfe1ea90890c5468ae2ff8aa1a0a85cb546e641ef8f2588cfad3b9cc
>
>
>This review also had similar unexplained slowness, but has already
>been rechecked by someone else and now passes. I note that the
>slowness in both cases was from the same TH worker node, and I will
>keep an eye on that node today.
>
>https://urldefense.proofpoint.com/v1/url?u=https://review.openstack.org/%2
>3/c/56420&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=eH0pxTUZo8NPZyF6hgoMQu%2Bf
>Dtysg45MkPhCZFxPEq8%3D%0A&m=QjlZEwdZOsbG6HW%2FAwjBP1GxhEprFcAcOx%2Btg9BHc4
>Q%3D%0A&s=4a4948f494e74e51749c2d43571071cdeaaef367a00a2f5ceca2ac06d2a0e862
>
>
>This review also had slowness in migration 206, but has been rechecked
>by the developer and now passes. It wasn't on the percona-001 worker
>that the other two were on, so perhaps this indicates that we need to
>relax the timing requirements for migration 206.
>
>Hope this helps,
>Michael
>
>On Wed, Jan 1, 2014 at 12:34 AM, Gary Kotton  wrote:
>> Hi,
>> It seems that she/he is behaving oddly again. I have posted a patch that
>> does not have any database changes and it has give me a ­1Š.
>> Happy new year
>> Gary
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> 
>>https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi
>>-bin/mailman/listinfo/openstack-dev&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r
>>=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZFxPEq8%3D%0A&m=QjlZEwdZOsbG6HW%2F
>>AwjBP1GxhEprFcAcOx%2Btg9BHc4Q%3D%0A&s=8ef5e4e15445067795d555001b6b8bdb4de
>>8575c4691f682db06db4f35bce5fe
>>
>
>
>
>-- 
>Rackspace Australia
>
>___
>OpenStack-dev mailing list
>OpenStack-dev@lists.openstack.org
>https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi-
>bin/mailman/listinfo/openstack-dev&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=e
>H0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZFxPEq8%3D%0A&m=QjlZEwdZOsbG6HW%2FAwj
>BP1GxhEprFcAcOx%2Btg9BHc4Q%3D%0A&s=8ef5e4e15445067795d555001b6b8bdb4de8575
>c4691f682db06db4f35bce5fe


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Turbo-hipster

[Michael Still]

Hi.

So while turbo hipster is new, I've been reading every failure message
it produces to make sure its not too badly wrong. There were four
failures posted last night while I slept:

https://review.openstack.org/#/c/64521


This one is a TH bug. We shouldn't be testing stable branches.
bug/1265238 has been filed to track this.

https://review.openstack.org/#/c/61753


This is your review. The failed run's log is
https://ssl.rcbops.com/turbo_hipster/logviewer/?q=/turbo_hipster/results/61/61753/8/check/gate-real-db-upgrade_nova_percona_user_001/1326092/user_001.log
and you can see from the failure message that migrations 152 and 206
took "too long".

Migration 152 took 326 seconds, where our historical data of 2,846
test migrations says it should take 222 seconds. Migration 206 took 81
seconds, where we think it should take 56 seconds based on 2,940 test
runs.

Whilst I can't explain why those migrations took too long this time
around, they are certainly exactly the sort of thing TH is meant to
catch. If you think your patch isn't responsible (perhaps the machine
is just being slow or something), you can always retest by leaving a
review comment of "recheck migrations". I have done this for you on
this patch.

https://review.openstack.org/#/c/61717


This review also had similar unexplained slowness, but has already
been rechecked by someone else and now passes. I note that the
slowness in both cases was from the same TH worker node, and I will
keep an eye on that node today.

https://review.openstack.org/#/c/56420


This review also had slowness in migration 206, but has been rechecked
by the developer and now passes. It wasn't on the percona-001 worker
that the other two were on, so perhaps this indicates that we need to
relax the timing requirements for migration 206.

Hope this helps,
Michael

On Wed, Jan 1, 2014 at 12:34 AM, Gary Kotton  wrote:
> Hi,
> It seems that she/he is behaving oddly again. I have posted a patch that
> does not have any database changes and it has give me a –1….
> Happy new year
> Gary
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Rackspace Australia

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev