Re: [openstack-dev] [keystone][performance][profiling] Profiling Mitaka Keystone: some results and asking for a help

[Dina Belova]

Thank you Morgan for quick fixes proposed!

On Tue, Apr 12, 2016 at 6:19 PM, Morgan Fainberg 
wrote:

> Sorry Missed the copy/paste of links:
>
> https://bugs.launchpad.net/keystone/+bug/1567403 [0]
> https://bugs.launchpad.net/keystone/+bug/1567413 [1]
>
> [0]
> https://review.openstack.org/#/q/I4857cfe1e62d54c3c89a0206ffc895c4cf681ce5,n,z
> [1] https://review.openstack.org/#/c/304688/
>
> --Morgan
>
> On Tue, Apr 12, 2016 at 8:16 AM, Morgan Fainberg <
> morgan.fainb...@gmail.com> wrote:
>
>> Fixes have been proposed for both of these bugs.
>>
>> Cheers,
>> --Morgan
>>
>> On Tue, Apr 12, 2016 at 12:38 AM, Dina Belova 
>> wrote:
>>
>>> Matt,
>>>
>>> Thanks for sharing the information about your benchmark. Indeed we need
>>> to follow up on this topic (I'll attend the summit). Let's try to collect
>>> as much information as possible prior Austin to have more facts to operate.
>>> I'll try to figure out why local context cache did not work at least on my
>>> environment, and, due to the results, most probably it did not act as
>>> supposed during your benchmarking as well.
>>>
>>> Cheers,
>>> Dina
>>>
>>> On Mon, Apr 11, 2016 at 10:57 PM, Matt Fischer 
>>> wrote:
>>>
 On Mon, Apr 11, 2016 at 8:11 AM, Dina Belova 
 wrote:

> Hey, openstackers!
>
> Recently I was trying to profile Keystone (OpenStack Liberty vs
> Mitaka) using this set of changes
> 
>  (that's
> currently on review - some final steps are required there to finish the
> work) and OSprofiler.
>
> Some preliminary results (all in one OpenStack node) can be found here
> 
>  (raw
> OSprofiler reports are not yet merged to some place and can be found
> here ). The full plan
> 
>  of
> what's going to be tested  can be found in the docs as well. In short I
> wanted to take a look how does Keystone changed its DB/Cache usage from
> Liberty to Mitaka, keeping in mind that there were several changes
> introduced:
>
>- federation support was added (and made DB scheme a bit more
>complex)
>- Keystone moved to oslo.cache usage
>- local context cache was introduced during Mitaka
>
> First of all - *good job on making Keystone less DB-extensive in case
> of cache turned on*! If Keystone caching is turned on, number of DB
> queries done to Keystone DB in Mitaka is averagely twice less than in
> Liberty, comparing the same requests and topologies. Thanks Keystone
> community to make it happen :)
>
> Although, I faced *two strange issues* during my experiments, and I'm
> kindly asking you, folks, to help me here:
>
>- I've created #1567403
> bug to share
>information - when I turned caching on, local context cache should 
> cache
>identical per API requests function calls not to ping Memcache too 
> often.
>Although I faced such calls, Keystone still used Memcache to gather 
> this
>information. May someone take a look on this and help me figure out 
> what am
>I observing? At the first sight local context cache should work ok, 
> but for
>some reason I do not see it's being used.
>- One more filed bug - #1567413
> - is about a
>bit opposite situation :) When I turned cache off explicitly in the
>keystone.conf file, I still observed some of the values being fetched 
> from
>Memcache... Your help is very appreciated!
>
> Thanks in advance and sorry for a long email :)
>
> Cheers,
> Dina
>
>
 Dina,

 Thanks for starting this conversation. I had some weird perf results
 comparing L to an RC release of Mitaka, but I was holding them until
 someone else confirmed what I saw. I'm testing token creation and
 validation. From what I saw, token validation slowed down in Mitaka. After
 doing my benchmark runs, the traffic to memcache was 8x in Mitaka from what
 it was in Liberty. That implies more caching but 8x is a lot and even
 memcache references are not free.

 I know some of the Keystone folks are looking into this so it will be
 good to follow-up on it. Maybe we could talk about this at the summit?




 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-

Re: [openstack-dev] [keystone][performance][profiling] Profiling Mitaka Keystone: some results and asking for a help

[Morgan Fainberg]

Sorry Missed the copy/paste of links:

https://bugs.launchpad.net/keystone/+bug/1567403 [0]
https://bugs.launchpad.net/keystone/+bug/1567413 [1]

[0]
https://review.openstack.org/#/q/I4857cfe1e62d54c3c89a0206ffc895c4cf681ce5,n,z
[1] https://review.openstack.org/#/c/304688/

--Morgan

On Tue, Apr 12, 2016 at 8:16 AM, Morgan Fainberg 
wrote:

> Fixes have been proposed for both of these bugs.
>
> Cheers,
> --Morgan
>
> On Tue, Apr 12, 2016 at 12:38 AM, Dina Belova 
> wrote:
>
>> Matt,
>>
>> Thanks for sharing the information about your benchmark. Indeed we need
>> to follow up on this topic (I'll attend the summit). Let's try to collect
>> as much information as possible prior Austin to have more facts to operate.
>> I'll try to figure out why local context cache did not work at least on my
>> environment, and, due to the results, most probably it did not act as
>> supposed during your benchmarking as well.
>>
>> Cheers,
>> Dina
>>
>> On Mon, Apr 11, 2016 at 10:57 PM, Matt Fischer 
>> wrote:
>>
>>> On Mon, Apr 11, 2016 at 8:11 AM, Dina Belova 
>>> wrote:
>>>
 Hey, openstackers!

 Recently I was trying to profile Keystone (OpenStack Liberty vs Mitaka)
 using this set of changes
 
  (that's
 currently on review - some final steps are required there to finish the
 work) and OSprofiler.

 Some preliminary results (all in one OpenStack node) can be found here
 
  (raw
 OSprofiler reports are not yet merged to some place and can be found
 here ). The full plan
 
  of
 what's going to be tested  can be found in the docs as well. In short I
 wanted to take a look how does Keystone changed its DB/Cache usage from
 Liberty to Mitaka, keeping in mind that there were several changes
 introduced:

- federation support was added (and made DB scheme a bit more
complex)
- Keystone moved to oslo.cache usage
- local context cache was introduced during Mitaka

 First of all - *good job on making Keystone less DB-extensive in case
 of cache turned on*! If Keystone caching is turned on, number of DB
 queries done to Keystone DB in Mitaka is averagely twice less than in
 Liberty, comparing the same requests and topologies. Thanks Keystone
 community to make it happen :)

 Although, I faced *two strange issues* during my experiments, and I'm
 kindly asking you, folks, to help me here:

- I've created #1567403
 bug to share
information - when I turned caching on, local context cache should cache
identical per API requests function calls not to ping Memcache too 
 often.
Although I faced such calls, Keystone still used Memcache to gather this
information. May someone take a look on this and help me figure out 
 what am
I observing? At the first sight local context cache should work ok, but 
 for
some reason I do not see it's being used.
- One more filed bug - #1567413
 - is about a bit
opposite situation :) When I turned cache off explicitly in the
keystone.conf file, I still observed some of the values being fetched 
 from
Memcache... Your help is very appreciated!

 Thanks in advance and sorry for a long email :)

 Cheers,
 Dina


>>> Dina,
>>>
>>> Thanks for starting this conversation. I had some weird perf results
>>> comparing L to an RC release of Mitaka, but I was holding them until
>>> someone else confirmed what I saw. I'm testing token creation and
>>> validation. From what I saw, token validation slowed down in Mitaka. After
>>> doing my benchmark runs, the traffic to memcache was 8x in Mitaka from what
>>> it was in Liberty. That implies more caching but 8x is a lot and even
>>> memcache references are not free.
>>>
>>> I know some of the Keystone folks are looking into this so it will be
>>> good to follow-up on it. Maybe we could talk about this at the summit?
>>>
>>>
>>>
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>>
>> --
>>
>> Best regards,
>>
>> Dina Belova
>>
>> Software Engineer
>>
>> Mirantis Inc.
>>
>> __
>> OpenStack Development Mailing List (not for usage qu

Re: [openstack-dev] [keystone][performance][profiling] Profiling Mitaka Keystone: some results and asking for a help

[Morgan Fainberg]

Fixes have been proposed for both of these bugs.

Cheers,
--Morgan

On Tue, Apr 12, 2016 at 12:38 AM, Dina Belova  wrote:

> Matt,
>
> Thanks for sharing the information about your benchmark. Indeed we need to
> follow up on this topic (I'll attend the summit). Let's try to collect as
> much information as possible prior Austin to have more facts to operate.
> I'll try to figure out why local context cache did not work at least on my
> environment, and, due to the results, most probably it did not act as
> supposed during your benchmarking as well.
>
> Cheers,
> Dina
>
> On Mon, Apr 11, 2016 at 10:57 PM, Matt Fischer 
> wrote:
>
>> On Mon, Apr 11, 2016 at 8:11 AM, Dina Belova 
>> wrote:
>>
>>> Hey, openstackers!
>>>
>>> Recently I was trying to profile Keystone (OpenStack Liberty vs Mitaka)
>>> using this set of changes
>>> 
>>>  (that's
>>> currently on review - some final steps are required there to finish the
>>> work) and OSprofiler.
>>>
>>> Some preliminary results (all in one OpenStack node) can be found here
>>> 
>>>  (raw
>>> OSprofiler reports are not yet merged to some place and can be found
>>> here ). The full plan
>>> 
>>>  of
>>> what's going to be tested  can be found in the docs as well. In short I
>>> wanted to take a look how does Keystone changed its DB/Cache usage from
>>> Liberty to Mitaka, keeping in mind that there were several changes
>>> introduced:
>>>
>>>- federation support was added (and made DB scheme a bit more
>>>complex)
>>>- Keystone moved to oslo.cache usage
>>>- local context cache was introduced during Mitaka
>>>
>>> First of all - *good job on making Keystone less DB-extensive in case
>>> of cache turned on*! If Keystone caching is turned on, number of DB
>>> queries done to Keystone DB in Mitaka is averagely twice less than in
>>> Liberty, comparing the same requests and topologies. Thanks Keystone
>>> community to make it happen :)
>>>
>>> Although, I faced *two strange issues* during my experiments, and I'm
>>> kindly asking you, folks, to help me here:
>>>
>>>- I've created #1567403
>>> bug to share
>>>information - when I turned caching on, local context cache should cache
>>>identical per API requests function calls not to ping Memcache too often.
>>>Although I faced such calls, Keystone still used Memcache to gather this
>>>information. May someone take a look on this and help me figure out what 
>>> am
>>>I observing? At the first sight local context cache should work ok, but 
>>> for
>>>some reason I do not see it's being used.
>>>- One more filed bug - #1567413
>>> - is about a bit
>>>opposite situation :) When I turned cache off explicitly in the
>>>keystone.conf file, I still observed some of the values being fetched 
>>> from
>>>Memcache... Your help is very appreciated!
>>>
>>> Thanks in advance and sorry for a long email :)
>>>
>>> Cheers,
>>> Dina
>>>
>>>
>> Dina,
>>
>> Thanks for starting this conversation. I had some weird perf results
>> comparing L to an RC release of Mitaka, but I was holding them until
>> someone else confirmed what I saw. I'm testing token creation and
>> validation. From what I saw, token validation slowed down in Mitaka. After
>> doing my benchmark runs, the traffic to memcache was 8x in Mitaka from what
>> it was in Liberty. That implies more caching but 8x is a lot and even
>> memcache references are not free.
>>
>> I know some of the Keystone folks are looking into this so it will be
>> good to follow-up on it. Maybe we could talk about this at the summit?
>>
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
>
> Best regards,
>
> Dina Belova
>
> Software Engineer
>
> Mirantis Inc.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone][performance][profiling] Profiling Mitaka Keystone: some results and asking for a help

[Dina Belova]

Matt,

Thanks for sharing the information about your benchmark. Indeed we need to
follow up on this topic (I'll attend the summit). Let's try to collect as
much information as possible prior Austin to have more facts to operate.
I'll try to figure out why local context cache did not work at least on my
environment, and, due to the results, most probably it did not act as
supposed during your benchmarking as well.

Cheers,
Dina

On Mon, Apr 11, 2016 at 10:57 PM, Matt Fischer  wrote:

> On Mon, Apr 11, 2016 at 8:11 AM, Dina Belova  wrote:
>
>> Hey, openstackers!
>>
>> Recently I was trying to profile Keystone (OpenStack Liberty vs Mitaka)
>> using this set of changes
>> 
>>  (that's
>> currently on review - some final steps are required there to finish the
>> work) and OSprofiler.
>>
>> Some preliminary results (all in one OpenStack node) can be found here
>> 
>>  (raw
>> OSprofiler reports are not yet merged to some place and can be found here
>> ). The full plan
>> 
>>  of
>> what's going to be tested  can be found in the docs as well. In short I
>> wanted to take a look how does Keystone changed its DB/Cache usage from
>> Liberty to Mitaka, keeping in mind that there were several changes
>> introduced:
>>
>>- federation support was added (and made DB scheme a bit more complex)
>>- Keystone moved to oslo.cache usage
>>- local context cache was introduced during Mitaka
>>
>> First of all - *good job on making Keystone less DB-extensive in case of
>> cache turned on*! If Keystone caching is turned on, number of DB queries
>> done to Keystone DB in Mitaka is averagely twice less than in Liberty,
>> comparing the same requests and topologies. Thanks Keystone community to
>> make it happen :)
>>
>> Although, I faced *two strange issues* during my experiments, and I'm
>> kindly asking you, folks, to help me here:
>>
>>- I've created #1567403
>> bug to share
>>information - when I turned caching on, local context cache should cache
>>identical per API requests function calls not to ping Memcache too often.
>>Although I faced such calls, Keystone still used Memcache to gather this
>>information. May someone take a look on this and help me figure out what 
>> am
>>I observing? At the first sight local context cache should work ok, but 
>> for
>>some reason I do not see it's being used.
>>- One more filed bug - #1567413
>> - is about a bit
>>opposite situation :) When I turned cache off explicitly in the
>>keystone.conf file, I still observed some of the values being fetched from
>>Memcache... Your help is very appreciated!
>>
>> Thanks in advance and sorry for a long email :)
>>
>> Cheers,
>> Dina
>>
>>
> Dina,
>
> Thanks for starting this conversation. I had some weird perf results
> comparing L to an RC release of Mitaka, but I was holding them until
> someone else confirmed what I saw. I'm testing token creation and
> validation. From what I saw, token validation slowed down in Mitaka. After
> doing my benchmark runs, the traffic to memcache was 8x in Mitaka from what
> it was in Liberty. That implies more caching but 8x is a lot and even
> memcache references are not free.
>
> I know some of the Keystone folks are looking into this so it will be good
> to follow-up on it. Maybe we could talk about this at the summit?
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 

Best regards,

Dina Belova

Software Engineer

Mirantis Inc.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone][performance][profiling] Profiling Mitaka Keystone: some results and asking for a help

[Matt Fischer]

On Mon, Apr 11, 2016 at 8:11 AM, Dina Belova  wrote:

> Hey, openstackers!
>
> Recently I was trying to profile Keystone (OpenStack Liberty vs Mitaka)
> using this set of changes
> 
>  (that's
> currently on review - some final steps are required there to finish the
> work) and OSprofiler.
>
> Some preliminary results (all in one OpenStack node) can be found here
> 
>  (raw
> OSprofiler reports are not yet merged to some place and can be found here
> ). The full plan
> 
>  of
> what's going to be tested  can be found in the docs as well. In short I
> wanted to take a look how does Keystone changed its DB/Cache usage from
> Liberty to Mitaka, keeping in mind that there were several changes
> introduced:
>
>- federation support was added (and made DB scheme a bit more complex)
>- Keystone moved to oslo.cache usage
>- local context cache was introduced during Mitaka
>
> First of all - *good job on making Keystone less DB-extensive in case of
> cache turned on*! If Keystone caching is turned on, number of DB queries
> done to Keystone DB in Mitaka is averagely twice less than in Liberty,
> comparing the same requests and topologies. Thanks Keystone community to
> make it happen :)
>
> Although, I faced *two strange issues* during my experiments, and I'm
> kindly asking you, folks, to help me here:
>
>- I've created #1567403
> bug to share
>information - when I turned caching on, local context cache should cache
>identical per API requests function calls not to ping Memcache too often.
>Although I faced such calls, Keystone still used Memcache to gather this
>information. May someone take a look on this and help me figure out what am
>I observing? At the first sight local context cache should work ok, but for
>some reason I do not see it's being used.
>- One more filed bug - #1567413
> - is about a bit
>opposite situation :) When I turned cache off explicitly in the
>keystone.conf file, I still observed some of the values being fetched from
>Memcache... Your help is very appreciated!
>
> Thanks in advance and sorry for a long email :)
>
> Cheers,
> Dina
>
>
Dina,

Thanks for starting this conversation. I had some weird perf results
comparing L to an RC release of Mitaka, but I was holding them until
someone else confirmed what I saw. I'm testing token creation and
validation. From what I saw, token validation slowed down in Mitaka. After
doing my benchmark runs, the traffic to memcache was 8x in Mitaka from what
it was in Liberty. That implies more caching but 8x is a lot and even
memcache references are not free.

I know some of the Keystone folks are looking into this so it will be good
to follow-up on it. Maybe we could talk about this at the summit?
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [keystone][performance][profiling] Profiling Mitaka Keystone: some results and asking for a help

[Dina Belova]

Hey, openstackers!

Recently I was trying to profile Keystone (OpenStack Liberty vs Mitaka)
using this set of changes

(that's
currently on review - some final steps are required there to finish the
work) and OSprofiler.

Some preliminary results (all in one OpenStack node) can be found here

(raw
OSprofiler reports are not yet merged to some place and can be found here
). The full plan

of
what's going to be tested  can be found in the docs as well. In short I
wanted to take a look how does Keystone changed its DB/Cache usage from
Liberty to Mitaka, keeping in mind that there were several changes
introduced:

   - federation support was added (and made DB scheme a bit more complex)
   - Keystone moved to oslo.cache usage
   - local context cache was introduced during Mitaka

First of all - *good job on making Keystone less DB-extensive in case of
cache turned on*! If Keystone caching is turned on, number of DB queries
done to Keystone DB in Mitaka is averagely twice less than in Liberty,
comparing the same requests and topologies. Thanks Keystone community to
make it happen :)

Although, I faced *two strange issues* during my experiments, and I'm
kindly asking you, folks, to help me here:

   - I've created #1567403
    bug to share
   information - when I turned caching on, local context cache should cache
   identical per API requests function calls not to ping Memcache too often.
   Although I faced such calls, Keystone still used Memcache to gather this
   information. May someone take a look on this and help me figure out what am
   I observing? At the first sight local context cache should work ok, but for
   some reason I do not see it's being used.
   - One more filed bug - #1567413
    - is about a bit
   opposite situation :) When I turned cache off explicitly in the
   keystone.conf file, I still observed some of the values being fetched from
   Memcache... Your help is very appreciated!

Thanks in advance and sorry for a long email :)

Cheers,
Dina
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev