[Openstack] [keystone] memcache token backend performance

[Xu (Simon) Chen]

Hi folks,

I am having trouble with using memcache as the keystone token backend. I
have three keystone nodes running active/active. Each is running keystone
on apache (for kerberos auth). I recently switched from using sql backend
to memcache, while have memcached running on all three of the keystone
nodes.

This setup would run well for a while, but then apache would start to hog
CPUs, and memcached would increase to 30% or so. I tried to increase
memcached cluster from 3 to 6 nodes, but in general the performance is much
worse compared to sql backend.

Any ideas?

Thanks.
-Simon
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

[Openstack] OpenStack Community Weekly Newsletter (Jan 3 2014)

[Stefano Maffulli]

  OpenStack Documentation Wrap Up for 2013
  


It's that time of the new year to reflect and look for ways to keep
improving the OpenStack docs. Here's a list of major events from 2013 in
OpenStack doc-land. Anne Gentle looks at the past year in review.


  OpenStack strong at LinuxConfAustralia (LCA) 

A full-day miniconf on Tuesday
 and a bunch of talks
in the main conference track:

  * Rapid OpenStack Deployment for Novices and Experts Alike by Florian
Haas 
  * Going Global: Building Global Clusters for OpenStack Swift by John
Dickinson 
  * Diskimage-builder: deep dive into a 'machine compiler' by Robert
Collins 
  * Provisioning Bare Metal with OpenStack by Devananda van der Veen

  * Systems Administration in the Open by Elizabeth Krumbach Joseph

  * Continuous Integration for your database migrations by Michael Still

  * How OpenStack Improves Code Quality with Project Gating and Zuul by
James Blair 
  * Processing Continuous Integration Log Events for Great Good by Clark
Boylan 

And there's at least one talk on third-party testing in the CI miniconf
on Monday  (schedule
still TBD), and maybe more surprises.


Tips 'n Tricks

  * By Cody Bunch
: OpenStack on
Raspberry Pi: Part 1 -- Why?

and OpenStack on Raspberry Pi: Part 2 -- Getting Started


and OpenStack on Raspberry Pi: Part 3 -- Packages



Upcoming Events

  * Project team leads report: Image, Telemetry, Block Storage
 Jan 07, 2014 --
Webinar Register 
  * OpenStack Online Meetup

Jan 09, 2014 -- Webinar Details

  * Project team leads report: Orchestration, Dashboard
 Jan 09, 2014 --
Webinar Register 
  * 3rd OpenStack User Group Nordics meetup

Jan 15, 2014 -- Stockholm, Sweden Details

  * Austin OpenStack User Group Meetup
 Jan 22,
2014 -- Austin, Texas Details

  * FOSDEM  Feb 01 -- 02, 2014 -- Bruxelles,
Belgium Details 
  * Austin OpenStack User Group Meetup
Feb 19,
2014 -- Austin, Texas Details



Reports from previous events

  * Russell Bryant, Compute (Nova) and John Dickinson, Object Storage
(Swift)




Other News

  * OpenStack RDO test day Jan 7-8

  * OpenMic Spotlight: Sergey Lukjanov

  * VMware Vs. KVM: OpenStack Hypervisor Race Heats Up


  * Keystone performance work

  * Murano 0.4 Brings You Metadata Repository, Neutron Support


  * OpenStack Project Meeting: not yet


Got Answers?

Ask OpenStack  is the go-to destination for
OpenStack users. Interesting questions waiting for answers:

  * Allow non-assigned ipaddresses as src on specific ports


 

Re: [Openstack] Monitoring OpenStack platform

[Tim Bell]

You could also have a look inside RDO (http://openstack.redhat.com). There are 
a set of Nagios probes inside the RPMs which run list commands against the 
servers to probe.

$ ls /usr/lib64/nagios/plugins/
check_nrpe  cinder-listglance-indexnegate swift-list  utils.sh
check_ping  eventhandlers  keystone-user-list  nova-list  urlize

Tim

From: Kaushal Shriyan [mailto:kaushalshri...@gmail.com]
Sent: 03 January 2014 19:07
To: Matt Kassawara
Cc: openstack
Subject: Re: [Openstack] Monitoring OpenStack platform


On Fri, Jan 3, 2014 at 10:56 AM, Matt Kassawara 
mailto:mkassaw...@gmail.com>> wrote:
Zenoss offers a module for OpenStack...

http://wiki.zenoss.org/ZenPack:OpenStack_Cloud_Monitor


Thanks Matt for the reply. Are there any similar features available in Nagios 
which is also a open source monitoring tool and the widely used. Any 
suggestions?

Regards,

Kaushal
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Monitoring OpenStack platform

[Tim Bell]

There is also some good advice in the logging and monitoring chapter of the 
OpenStack operations guide at 
http://docs.openstack.org/trunk/openstack-ops/content/logging_monitoring.html

Tim

From: Matt Kassawara [mailto:mkassaw...@gmail.com]
Sent: 03 January 2014 18:57
To: Kaushal Shriyan
Cc: openstack
Subject: Re: [Openstack] Monitoring OpenStack platform

Zenoss offers a module for OpenStack...

http://wiki.zenoss.org/ZenPack:OpenStack_Cloud_Monitor

On Fri, Jan 3, 2014 at 10:08 AM, Kaushal Shriyan 
mailto:kaushalshri...@gmail.com>> wrote:
Hi,

Are there monitoring tools to monitor OpenStack platform for example KVM 
Hypervisor and the guest hosts?

Regards,

Kaushal

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : 
openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Monitoring OpenStack platform

[Kaushal Shriyan]

On Fri, Jan 3, 2014 at 10:56 AM, Matt Kassawara wrote:

> Zenoss offers a module for OpenStack...
>
> http://wiki.zenoss.org/ZenPack:OpenStack_Cloud_Monitor
>
>
>
Thanks Matt for the reply. Are there any similar features available in
Nagios which is also a open source monitoring tool and the widely used. Any
suggestions?

Regards,

Kaushal
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

[Openstack] instance log

[Peter Cheung]

Hi AllI am designing the log window 
(http://peter.kingofcoders.com/wp-content/uploads/2014/01/log-window-doesnt-contain-all-logs.png)
 for titan to show the error log for an instance. When the instance is fail to 
start, i can use "nova show" to get the error message, but the message is only 
the last record of table instance_faults, so the log message is not complete. 
Any openstack command can retrieve all the log record from table 
instance_faults for a specific instance?

Thanksfrom Peter  ___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] this can crash devstack

[Peter Cheung]

Thanks Joshua
from Peter

> From: harlo...@yahoo-inc.com
> To: s...@dague.net; dtro...@gmail.com; mcheun...@hotmail.com
> CC: openstack@lists.openstack.org
> Subject: Re: [Openstack] this can crash devstack
> Date: Thu, 2 Jan 2014 18:01:15 +
> 
> If u want to try an alternate: http://anvil.readthedocs.org/
> 
> It goes through various stages:
> 
> 1. Downloading & preparing the components (git cloning...)
> 2. Building components rpms (and component dependencies to rpms)
>a. This creates 2 yum repositories that can be used for other installs
> 
> 3. Installing component rpms (and dependents created in #2)
> 4. Starting (uses the rpms installed services, aka `service nova-compute
> start`)
> 5. And the one u want restarting!
> 
> Other actions @ 
> https://github.com/stackforge/anvil/tree/master/anvil/actions
> 
> Its also not mean for deployment (although the created yum repos can be
> used for this) but it might get u farther and suite your end-goal better.
> 
> -Josh
> 
> On 1/2/14, 5:01 AM, "Sean Dague"  wrote:
> 
> >On 01/01/2014 11:16 PM, Dean Troyer wrote:
> >> On Wed, Jan 1, 2014 at 8:58 PM, Peter Cheung  >> > wrote:
> >> 
> >> 1) import an qcow image to glance
> >> 2) start the instance for that image
> >> 3) init 6
> >> 4) delete that qcow image
> >> 5) rejoin the devstack
> >> 
> >> devstack will still running, but unable to start any instance, even
> >> instances are not from that image.
> >> 
> >> 
> >> Restarting after a reboot is really outside what DevStack is intended to
> >> be able to do.  rejoin_stack.sh is only meant to restart the screen
> >> sessions and nothing else.  Even then it isn't perfect, which is why you
> >> should run stack.sh again after rebooting.
> >
> >Right, devstack is not a generic installer, it's a development tool.
> >Surviving reboot is intentionally not a solved problem, because that's
> >not what it's for.
> >
> > -Sean
> >
> >-- 
> >Sean Dague
> >Samsung Research America
> >s...@dague.net / sean.da...@samsung.com
> >http://dague.net
> >
> 
  ___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Monitoring OpenStack platform

[Matt Kassawara]

Zenoss offers a module for OpenStack...

http://wiki.zenoss.org/ZenPack:OpenStack_Cloud_Monitor


On Fri, Jan 3, 2014 at 10:08 AM, Kaushal Shriyan
wrote:

> Hi,
>
> Are there monitoring tools to monitor OpenStack platform for example KVM
> Hypervisor and the guest hosts?
>
> Regards,
>
> Kaushal
>
> ___
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

[Openstack] Monitoring OpenStack platform

[Kaushal Shriyan]

Hi,

Are there monitoring tools to monitor OpenStack platform for example KVM
Hypervisor and the guest hosts?

Regards,

Kaushal
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Ask] Cinder Ubuntu Saucy 13.10

[Swapnil Kulkarni]

I think instructions in [1] are sufficient. You can also confirm the same
on [2]


[1]
http://docs.openstack.org/trunk/install-guide/install/apt/content/basics-packages.html
[2] https://wiki.ubuntu.com/ServerTeam/CloudArchive

Best Regards,
Swapnil


On Fri, Jan 3, 2014 at 3:19 PM, Mahardhika Gilang <
mahardika.gil...@andalabs.com> wrote:

>  Hi all,
> i have done some configuration and all work except cinder, i used Ubuntu
> Saucy 13.10 in all physical machine. i've got cinder version 1.0.6
> is there you guys get this version too in 13.10 ubuntu? and i wondering,
> what is your apt repository or source for updating havana cloud?
>
> I just want make this clear, since i see many still used precise Ubuntu
> for cloud.
> --
> Regards,
> Mahardhika Gilang
>
>
> ___
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] How to achieve HA and Scalability for Neutron

[Li Ma]

> Right, the Neutron L3 agent is the only OpenStack service that is
> not stateless, and therefore you cannot use traditional
> load-balancing across a set of identical nodes.

Hi jay, how about dhcp-agent? I've went through the bps of neutron
for icehouse release. There's no related topics on dhcp-agent.

I wonder how to implement the same functionality of dhcp-agent.

Thanks very much,

Li Ma (Nick)
Email: skywalker.n...@gmail.com


2013/12/28 Li Ma :
> That's awesome!! I'll try and test the patch for grizzly and also the
> new function of icehouse.
>
> Many thanks, jay.
>
>
> 2013/12/27 Jay Pipes :
>> On 12/26/2013 02:22 PM, Diego Parrilla Santamaría wrote:
>>>
>>> That's really a nice stuff...
>>
>>
>> So, to wrap up this particular mini-thread... here is the solution:
>>
>> If on Grizzly or Havana and you don't want use Pacemaker (so that you can
>> spread L3 agent load across multiple nodes):
>>
>> 1) Apply this patch to Neutron:
>>
>> https://gist.github.com/jaypipes/8135839
>>
>> 2) Set the router_scheduler_driver in nova.conf to
>> neutron.scheduler.l3_agent_scheduler.LeastUtilizedScheduler
>>
>> Reference:
>> https://github.com/stackforge/cookbook-openstack-network/blob/master/templates/default/neutron.conf.erb#L232
>>
>> 3) Put this script into a cron job:
>>
>> https://github.com/stackforge/cookbook-openstack-network/blob/master/files/default/neutron-ha-tool.py
>>
>> If you're on Icehouse, simply do:
>>
>> 1) Set the router_scheduler_driver in nova.conf to
>> neutron.scheduler.l3_agent_scheduler.LeastRoutersScheduler
>>
>> Reference:
>> https://github.com/openstack/neutron/blob/master/etc/neutron.conf#L223
>>
>> 2) Put this script into a cron job:
>>
>> https://github.com/stackforge/cookbook-openstack-network/blob/master/files/default/neutron-ha-tool.py
>>
>> Best,
>> -jay
>>
>>> On Thu, Dec 26, 2013 at 5:58 PM, Jay Pipes >> > wrote:
>>>
>>> On 12/26/2013 07:43 AM, skywalker.n...@gmail.com
>>>  wrote:
>>>
>>> Hi stackers,
>>> I'm preparing for an OpenStack cloud in the production system.
>>> I've
>>> achieved HA and load balancing for most of the components to a
>>> certain
>>> degree.
>>> Finally I find it hard for Neutron to do such configuration. Are
>>> there
>>> any ways to run multiple l3-agents and dhcp-agents and cluster
>>> them
>>> together?
>>>
>>>
>>> Right, the Neutron L3 agent is the only OpenStack service that is
>>> not stateless, and therefore you cannot use traditional
>>> load-balancing across a set of identical nodes.
>>>
>>> That said, there's nothing wrong with running multiple L3 nodes,
>>> with routers for different tenants hosted on different L3 agents. We
>>> do this successfully in our deployment using a custom Neutron
>>> scheduler that my colleague Alan Meadows wrote [1] and a Python
>>> script (also written by Alan) that runs in cron looking for failures
>>> on an L3 agent and if found, moves the routers from the failed node
>>> to a working one. [2]
>>>
>>> The advantage to this vs. something like pacemaker is that you
>>> spread the L3 agent workload across many nodes -- accomplishing a
>>> sort of poor-man's load balancing/sharding for L3 agent requests.
>>>
>>> Best,
>>> -jay
>>>
>>> [1] https://gist.github.com/__jaypipes/8135839
>>> 
>>> [2]
>>>
>>> https://github.com/stackforge/__cookbook-openstack-network/__blob/master/files/default/__neutron-ha-tool.py
>>>
>>> 
>>>
>>>
>>> _
>>> Mailing list:
>>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack
>>>
>>> 
>>> Post to : openstack@lists.openstack.org
>>> 
>>> Unsubscribe :
>>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack
>>> 
>>>
>>>
>>
>>
>> ___
>> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to : openstack@lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
>
> --
> Li Ma (Nick)
>
> Architect of Stackinsider
>
> Email: skywalker.n...@gmail.com

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

[Openstack] [Ask] Cinder Ubuntu Saucy 13.10

[Mahardhika Gilang]

Hi all,
i have done some configuration and all work except cinder, i used Ubuntu 
Saucy 13.10 in all physical machine. i've got cinder version 1.0.6
is there you guys get this version too in 13.10 ubuntu? and i wondering, 
what is your apt repository or source for updating havana cloud?


I just want make this clear, since i see many still used precise Ubuntu 
for cloud.

--
Regards,
Mahardhika Gilang

**
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Keystone] Keystone performance work

[Joshua Harlow]

Awesome!!

I am looking forward to any results u get. For said results when they arrive 
make sure they are reproducible and accurate, and then people can start solving 
any performance problems in a way that can be compared to the initial 
baseline...

2014 the year of rally :P

Sent from my really tiny device...

On Jan 2, 2014, at 11:32 PM, "Neependra Khare" 
mailto:nkh...@redhat.com>> wrote:

Hi All,

I wanted to share the current status of this work.

Thanks to Boris and the team for implementing keystone benchmark with Rally.
It has been merged with upstream Rally code base couple of days back.

I have tested above with devstack on Fedora 19 and Ubuntu 12.04.

With Rally Deploy engine we can deploy OpenStack distribution like DevStack
before running the benchmark.
https://wiki.openstack.org/wiki/Rally/DeployEngines#DevstackEngine

I am planning to prepare the localrc file for devstack by which we can configure
keystone with different configuration like with PKI/UUID tokens, with or without
memcached etc. This would help us is automating the process. I would be
documenting everything and would share it with the community.

Regards,
Neependra


On 12/20/2013 02:09 PM, Neependra Khare wrote:
Hi Joshua,

On 12/20/2013 01:34 PM, Joshua Harlow wrote:
Awesome i will see if I can make it, maybe u guys can send out a summary 
afterwards? Glad to see performance work (and the associated tooling) get the 
attention it deserves!
We had a meeting yesterday. Etherpad has most of things we discussed:-
https://etherpad.openstack.org/p/Rally_Keystone_Benchmarks

We decided to have specific pattern in workload (like user name) for keystone
benchmarks as that would make clean-up easier.

INIT methods like creating N users, would setup environment before running the
actual setup.  Jay suggested set that may be we can use some form of snapshot
for images, which have per-populated dataset. So that, we do not have to spend
time in INIT methods every time we run the tests.

Next steps
- Boris would have the clean-up and basic INIT functionality written down for 
Keystone
, which would then help us to write our first Keystone benchmark.
- Meanwhile we'll update the benchmark we want to run with Rally in "List of 
benchmarks
that we would like to have" section of the etherpad.

Boris, Jay, Hugh, Tristan and others,
If I missed any important point then please feel free to add here.

Regards,
Neependra



Sent from my really tiny device...

On Dec 18, 2013, at 5:32 AM, "Neependra Khare" 
mailto:nkh...@redhat.com>> wrote:

On 12/17/2013 05:19 PM, Neependra Khare wrote:
On 12/17/2013 12:48 AM, Joshua Harlow wrote:

Another thing that u might consider.

The rally project[1] has been using a tool called tomograph[2] and making
tomograph better and it has been collecting some similar use-cases and
test-cases around various openstack performance related work (also it has
been working on defining said methodology, how to setup for performance
tests, and more...).

Some examples of various keystone/nova/glance related calls (see the low
level trace information gathered there):


Thanks Josh.
We have started a blueprint to have keystone related benchmarks
with Rally.
https://blueprints.launchpad.net/rally/+spec/keystone-benchmark
We are planning to have g+ hangout tomorrow (Dec 19, 2013)  to discuss
Keystone related benchmarks with Rally at 11:00 EST
https://etherpad.openstack.org/p/Rally_Keystone_Benchmarks

Feel free to join it. Just update your name on above mentioned etherpad
under Interested participants section.

Regards,
Neependra



Regards,
Neependra

-
https://raw.github.com/stackforge/tomograph/master/doc/screenshots/zipkin-g
lance-image-list.png
- http://www.slideshare.net/mirantis/rally-benchmarkingatscale/24

It would seem like some of our scripts/test-cases would actually fit quite
nicely into rally instead of being one-offs.

I know that boris-42 would appreciate a single solution instead of
one-offs. It seems like rally is becoming that.

[1] https://wiki.openstack.org/wiki/Rally
[2] https://github.com/stackforge/tomograph

Jump in IRC and the rally team I'm sure would love to chat
(#openstack-rally, freenode).

-Josh

On 12/16/13, 10:00 AM, "Jay Pipes" 
 wrote:



On 12/16/2013 02:25 AM, Neependra Khare wrote:


Hi Jay,

Thanks for your comments.  Please find my reply in-line.

On 12/14/2013 12:58 AM, Jay Pipes wrote:


I have listed down the methodology I'll be following for this test:-


https://wiki.openstack.org/wiki/KeystonePerformance#Identify_CPU.2C_Dis
k.2C_Memory.2C_Database_bottlenecks



My first suggestion would be to rework the performance benchmarking
work items to have clearer indications regarding *what are the metrics
being tested* in each work item.


Performance characterization is an iterative process. I am open to
rework on the work-items as we
go along.


Right, but the smaller the work item, the easier the iterations are :)



For example, the first work item is "Identi