Re: [Openstack-operators] [openstack-dev] [QA][LCOO] MEX-ops-meetup: OpenStack Extreme Testing

[Sam P]

Hi Boris,

 Thanks for comment.
 Sorry for the misleading context in the etherpad. It sounds like Eris
will do everything, but it is not.
 I think I need to add more details to etherpad (I will do that)
 Short answer is we are not into  "reinventing wheels" and we will use
all possible existing tools.
 # Me and Gautam are working on a document about what are the gaps,
and what tool should use on what purpose and why.
 # I think that would be the long answer and  we definitely need your
feed back on this.
 #  I will share that here..

 For, Rally,
  Actually we are using Rally in Eris. Here is the PoC we are working on (WIP).
 # At this point, PoC is in it's very early stage. Not all the
Concepts are implemented yet..
  code: https://github.com/gautamdivgi/eris
  doc: 
https://openstack-lcoo.atlassian.net/wiki/spaces/LCOO/pages/22872097/Extreme+Testing+Demo

 You may find lot more doc's about Eris in here,
 
https://openstack-lcoo.atlassian.net/wiki/spaces/LCOO/pages/13393034/Eris+-+Extreme+Testing+Framework+for+OpenStack

 For an example, we are considering,
 Rally: For control plane load injection (Current PoC we don't use Rally
 Shaker: For data plane load injection
 os-fault: fault injection ( lot of work need to be done here)

 Here is a one example:
 Eris is focus on realistic outage scenarios such as, SDN controller
failure, Storage back-end failure, infra L2/L3 SW failure ,etc,,
 If those are vendor specific appliances, then how should we do
failure injection and metrics gathering?
 Eris will provide plugin mechanism to plug vendor drivers ( provide
by the vendor).
 In testing, Eris will use Rally and Shaker for load generation and
inject deterministic/random HW/SW failures in those
 appliances and gather the metrics. Then,  Eris (or Rally or any other
tool or mix of them) can analyze the result and
 create the final report.

 Sorry for ranting.

--- Regards,
Sampath



On Wed, Nov 1, 2017 at 4:02 PM, Boris Pavlovic <bo...@pavlovic.me> wrote:
> Sam,
>
>>  Etherpad: https://etherpad.openstack.org/p/SYD-extreme-testing
>
>
> I really don't want to sound like a person that say use Rally my best ever
> project blablbab and other BS.
> I think that "reinventing wheels" approach is how humanity evolves and
> that's why I like this effort in any case.
>
> But really, I read carefully etherpad and I really see in description of
> Eris just plain Rally as is:
>
> - Rally allows you to create tests as YAML
> - Rally allows you to inject in various actions during the load (Rally
> Hooks) which makes it easy to do chaos and other kind of testing
> - Rally is pluggable and you can write even your own Runners (scenario
> executors) that will generate load pattern that you need
> - Rally has SLAs plugins (that can deeply analyze result of test cases) and
> say whatever they pass or not
> - We are working on feature that allows you to mix different workloads in
> parallel (and generate more realistic load)
> -.
>
> So it would be really nice if you can share gaps that you faced that are
> blocking you to use directly Rally..
>
> Thanks!
>
> Best regards,
> Boris Pavlovic
>
>
> On Tue, Oct 31, 2017 at 10:50 PM, Sam P <sam47pr...@gmail.com> wrote:
>>
>> Hi All,
>>
>>  Sending out a gentle reminder of Sydney Summit Forum Session
>> regarding this topic.
>>
>>  Extreme/Destructive Testing
>>  Tuesday, November 7, 1:50pm-2:30pm
>>  Sydney Convention and Exhibition Centre - Level 4 - C4.11
>>
>> [https://www.openstack.org/summit/sydney-2017/summit-schedule/events/20470/extremedestructive-testing]
>>  Eatherpad: https://etherpad.openstack.org/p/SYD-extreme-testing
>>
>>  Your participation in this session would be greatly appreciated.
>> --- Regards,
>> Sampath
>>
>>
>>
>> On Mon, Aug 14, 2017 at 11:43 PM, Tim Bell <tim.b...@cern.ch> wrote:
>> > +1 for Boris’ suggestion. Many of us use Rally to probe our clouds and
>> > have
>> > significant tooling behind it to integrate with local availability
>> > reporting
>> > and trouble ticketing systems. It would be much easier to deploy new
>> > functionality such as you propose if it was integrated into an existing
>> > project framework (such as Rally).
>> >
>> >
>> >
>> > Tim
>> >
>> >
>> >
>> > From: Boris Pavlovic <bo...@pavlovic.me>
>> > Reply-To: "OpenStack Development Mailing List (not for usage questions)"
>> > <openstack-...@lists.openstack.org>
>> > Date: Monday, 14 August 2017 at 12:57
>> > To: "OpenStack Development Mailing List (not for usage questions)"
>> > <openstack-...@lists.openst

Re: [Openstack-operators] [openstack-dev] [QA][LCOO] MEX-ops-meetup: OpenStack Extreme Testing

[Sam P]

Hi All,

 Sending out a gentle reminder of Sydney Summit Forum Session
regarding this topic.

 Extreme/Destructive Testing
 Tuesday, November 7, 1:50pm-2:30pm
 Sydney Convention and Exhibition Centre - Level 4 - C4.11
 
[https://www.openstack.org/summit/sydney-2017/summit-schedule/events/20470/extremedestructive-testing]
 Eatherpad: https://etherpad.openstack.org/p/SYD-extreme-testing

 Your participation in this session would be greatly appreciated.
--- Regards,
Sampath



On Mon, Aug 14, 2017 at 11:43 PM, Tim Bell <tim.b...@cern.ch> wrote:
> +1 for Boris’ suggestion. Many of us use Rally to probe our clouds and have
> significant tooling behind it to integrate with local availability reporting
> and trouble ticketing systems. It would be much easier to deploy new
> functionality such as you propose if it was integrated into an existing
> project framework (such as Rally).
>
>
>
> Tim
>
>
>
> From: Boris Pavlovic <bo...@pavlovic.me>
> Reply-To: "OpenStack Development Mailing List (not for usage questions)"
> <openstack-...@lists.openstack.org>
> Date: Monday, 14 August 2017 at 12:57
> To: "OpenStack Development Mailing List (not for usage questions)"
> <openstack-...@lists.openstack.org>
> Cc: openstack-operators <openstack-operators@lists.openstack.org>
> Subject: Re: [openstack-dev] [QA][LCOO] MEX-ops-meetup: OpenStack Extreme
> Testing
>
>
>
> Sam,
>
>
>
> Seems like a good plan and huge topic ;)
>
>
>
> I would as well suggest to take a look at the similar efforts in OpenStack:
>
> - Failure injection: https://github.com/openstack/os-faults
>
> - Rally Hooks Mechanism (to inject in rally scenarios failures):
> https://rally.readthedocs.io/en/latest/plugins/implementation/hook_and_trigger_plugins.html
>
>
>
>
>
> Best regards,
> Boris Pavlovic
>
>
>
>
>
> On Mon, Aug 14, 2017 at 2:35 AM, Sam P <sam47pr...@gmail.com> wrote:
>
> Hi All,
>
> This is a follow up for OpenStack Extreme Testing session[1]
> we did in MEX-ops-meetup.
>
> Quick intro for those who were not there:
> In this work, we proposed to add new testing framework for openstack.
> This framework will provides tool for create tests with destructive
> scenarios which will check for High Availability, failover and
> recovery of OpenStack cloud.
> Please refer the link on top of the [1] for further details.
>
> Follow up:
> We are planning periodic irc meeting and have an irc
> channel for discussion. I will get back to you with those details soon.
>
> At that session, we did not have time to discuss last 3 items,
> Reference architectures
>  We are discussing about the reference architecture in [2].
>
> What sort of failures do you see today in your environment?
>  Currently we are considering, service failures, backend services (mq,
> DB, etc.) failures,
>  Network sw failures..etc. To begin with the implementation, we are
> considering to start with
>  service failures. Please let us know what failures are more frequent
> in your environment.
>
> Emulation/Simulation mechanisms, etc.
>  Rather than doing actual scale, load, or performance tests, we are
> thinking to build a emulation/simulation mechanism
> to get the predictions or result of how will openstack behave on such
> situations.
> This interesting idea was proposed by the Gautam and need more
> discussion on this.
>
> Please let us know you questions or comments.
>
> Request to Mike Perez:
>  We discussed about synergies with openstack assertion tags and other
> efforts to do similar testing in openstack.
>  Could you please give some info or pointer of previous discussions.
>
> [1] https://etherpad.openstack.org/p/MEX-ops-extreme-testing
> [2]
> https://openstack-lcoo.atlassian.net/wiki/spaces/LCOO/pages/15477787/Extreme+Testing-Vision+Arch
>
> --- Regards,
> Sampath
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [QA][LCOO] MEX-ops-meetup: OpenStack Extreme Testing

[Sam P]

Hi All,

This is a follow up for OpenStack Extreme Testing session[1]
we did in MEX-ops-meetup.

Quick intro for those who were not there:
In this work, we proposed to add new testing framework for openstack.
This framework will provides tool for create tests with destructive
scenarios which will check for High Availability, failover and
recovery of OpenStack cloud.
Please refer the link on top of the [1] for further details.

Follow up:
We are planning periodic irc meeting and have an irc
channel for discussion. I will get back to you with those details soon.

At that session, we did not have time to discuss last 3 items,
Reference architectures
 We are discussing about the reference architecture in [2].

What sort of failures do you see today in your environment?
 Currently we are considering, service failures, backend services (mq,
DB, etc.) failures,
 Network sw failures..etc. To begin with the implementation, we are
considering to start with
 service failures. Please let us know what failures are more frequent
in your environment.

Emulation/Simulation mechanisms, etc.
 Rather than doing actual scale, load, or performance tests, we are
thinking to build a emulation/simulation mechanism
to get the predictions or result of how will openstack behave on such
situations.
This interesting idea was proposed by the Gautam and need more
discussion on this.

Please let us know you questions or comments.

Request to Mike Perez:
 We discussed about synergies with openstack assertion tags and other
efforts to do similar testing in openstack.
 Could you please give some info or pointer of previous discussions.

[1] https://etherpad.openstack.org/p/MEX-ops-extreme-testing
[2] 
https://openstack-lcoo.atlassian.net/wiki/spaces/LCOO/pages/15477787/Extreme+Testing-Vision+Arch

--- Regards,
Sampath

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Remote pacemaker on coHi mpute nodes

[Sam P]

Hi,

 This might not what exactly you are looking for... but... you may extend this.
 In Masakari [0], we use pacemaker-remote in masakari-monitors[1] to
monitor node failures.
 In [1], there is hostmonitor.sh, which will gonna deprecate in next
cycle, but straightforward way to do this.
 [0] https://wiki.openstack.org/wiki/Masakari
 [1] 
https://github.com/openstack/masakari-monitors/tree/master/masakarimonitors/hostmonitor

 Then there is pacemaker-resources agents,
 https://github.com/openstack/openstack-resource-agents/tree/master/ocf

> I have already tried "pcs resource cleanup" but it cleans fine all resources
> but not remote nodes.
> Anycase on monday I'll send what you requested.
Hope we can get more details on Monday.

--- Regards,
Sampath



On Sat, May 13, 2017 at 9:52 PM, Ignazio Cassano
 wrote:
> Thanks Curtis.
> I have already tried "pcs resource cleanup" but it cleans fine all resources
> but not remote nodes.
> Anycase on monday I'll send what you requested.
> Regards
> Ignazio
>
> Il 13/Mag/2017 14:27, "Curtis"  ha scritto:
>
> On Fri, May 12, 2017 at 10:23 PM, Ignazio Cassano
>  wrote:
>> Hi Curtis, at this time I am using remote pacemaker only for controlli ng
>> openstack services on compute nodes (neutron openvswitch-agent,
>> nova-compute, ceilometer compute). I wrote my own ansible playbooks to
>> install and configure all components.
>> Second step could  be expand it for vm high availability.
>> I did not find any procedure for cleaning up compute node after rebooting
>> and I googled a lot without luck.
>
> Can you paste some putput of something like "pcs status" and I can try
> to take a look?
>
> I've only used pacemaker a little, but I'm fairly sure it's going to
> be something like "pcs resource cleanup "
>
> Thanks,
> Curtis.
>
>> Regards
>> Ignazio
>>
>> Il 13/Mag/2017 00:32, "Curtis"  ha scritto:
>>
>> On Fri, May 12, 2017 at 8:51 AM, Ignazio Cassano
>>  wrote:
>>> Hello All,
>>> I installed openstack newton p
>>> with a pacemaker cluster made up of 3 controllers and 2 compute nodes.
>>> All
>>> computer have centos 7.3.
>>> Compute nodes are provided with remote pacemaker ocf resource.
>>> If before shutting down a compute node I disable the compute node
>>> resource
>>> in the cluster and enable it when the compute returns up, it work fine
>>> and
>>> cluster shows it online.
>>> If the compute node goes down before disabling the compute node resource
>>> in
>>> the cluster, it remains offline also after it is powered up.
>>> The only solution I found is removing the compute node resource in the
>>> cluster and add it again with a different name (adding this new name in
>>> all
>>> controllers /etc/hosts file).
>>> With the above workaround it returns online for the cluster and all its
>>> resources  (openstack-nova-compute etc etc) return to work fine.
>>> Please,  does anyone know a better solution ?
>>
>> What are you using pacemaker for on the compute nodes? I have not done
>> that personally, but my impression is that sometimes people do that in
>> order to have virtual machines restarted somewhere else should the
>> compute node go down outside of a maintenance window (ie. "instance
>> high availability"). Is that your use case? If so, I would imagine
>> there is some kind of clean up procedure to put the compute node back
>> into use when pacemaker thinks it has failed. Did you use some kind of
>> openstack distribution or follow a particular installation document to
>> enable this pacemaker setup?
>>
>> It sounds like everything is working as expected (if my guess is
>> right) and you just need the right steps to bring the node back into
>> the cluster.
>>
>> Thanks,
>> Curtis.
>>
>>
>>> Regards
>>> Ignazio
>>>
>>>
>>> ___
>>> OpenStack-operators mailing list
>>> OpenStack-operators@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>>
>>
>>
>>
>> --
>> Blog: serverascode.com
>>
>>
>
>
>
> --
> Blog: serverascode.com
>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [HA] HA Discussion at Boston Forum

[Sam P]

Hi All,

This is a quick reminder for HA Forum session at Boston Summit.
Thank you all for your comments and effort to make this happen in Boston Summit.

 Time: Thu 11 , 11:00am-11:40am
 Location: Hynes Convention Center - Level One - MR 103
 Etherpad: https://etherpad.openstack.org/p/BOS-forum-HA-in-openstack

 Please join and let's discuss the HA issues in OpenStack...

--- Regards,
Sampath



On Mon, Apr 10, 2017 at 12:37 PM, Sam P <sam47pr...@gmail.com> wrote:
> Hi All,
>
>  I proposed a session [1] to Boston Forum and it is now at incomplete state.
>  This is good opportunity for all of us to get together and discuss
> our HA related issues.
>  I need your help to reform this session.
>  Please put comments on [1] or replay to this with your comments.
>
> [1] http://forumtopics.openstack.org/cfp/details/117
>
> --- Regards,
> Sampath

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [HA] HA Discussion at Boston Forum

[Sam P]

Hi All,

 I proposed a session [1] to Boston Forum and it is now at incomplete state.
 This is good opportunity for all of us to get together and discuss
our HA related issues.
 I need your help to reform this session.
 Please put comments on [1] or replay to this with your comments.

[1] http://forumtopics.openstack.org/cfp/details/117

--- Regards,
Sampath

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-community] OpenStack Summit Boston-CFP closes February 6th

[Sam P]

Hi Erin,

 Thank you for the reminder and other info.
 In you previous mail,
 > Panels are allowed a total of four speakers plus one moderator, whereas
 > presentations and lightning talks are limited to two speakers.
 To the best of my knowledge, presentations limited to three speakers.
 Not sure about the lightning talks.


--- Regards,
Sampath



On Thu, Feb 2, 2017 at 4:14 AM, Erin Disney  wrote:
> Hi Everyone-
>
> Don’t forget: the Call for Presentations for the upcoming OpenStack Summit
> Boston closes next week!
>
> The submission deadline is February 6, 2017 at 11:59PM PDT (February 7, 2017
> at 6:59 UTC).
>
> Reminder: Proposed sessions must indicate a format: Panel, presentation or
> lightning talk. Each format has a maximum number of speakers associated.
> Panels are allowed a total of four speakers plus one moderator, whereas
> presentations and lightning talks are limited to two speakers.
>
> As a reminder, speakers are limited to a maximum of three submissions total.
>
> Contact speakersupp...@openstack.org with any submission questions.
>
> BOSTON REGISTRATION
> Attendee registration is now open. Purchase your discounted early bird
> passes now. Prices will increase in mid March.
>
> SPONSORSHIP SALES
> Summit sponsorship sales are also open. You can now sign the electronic
> contract here. If you plan to sponsor both 2017 OpenStack Summits (Boston in
> May & Sydney in November), then check out page 4 of the Boston Summit
> Sponsorship Prospectus for a special 15% discount on Sydney Summit
> sponsorship packages. Please note this only applies to companies who sponsor
> both the Boston Summit and Sydney Summit. Full details of the sponsorship
> signing process are outlined here.
>
> If you have any general Summit questions, contact us at
> sum...@openstack.org.
>
>
> Erin Disney
> OpenStack Marketing
> e...@openstack.org
>
>
> ___
> Community mailing list
> commun...@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/community
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators