Re: [openstack-dev] [oslo_messaging] Limiting the number of retries for kafka driver

[Elancheran Subramanian]

Yes, there is a config option for *oslo_messaging_rabbit* 
default_notification_retry_attempts 
http://docs.openstack.org/mitaka/config-reference/compute/rpc.html


The problem with Nova and Neutron provide the retry mechanism, there is no 
configuration on nova or neutron, which can be passed onto the 
also_messaging. I will add a patch so that it would pick up from the 
configuration in oslo_messaging itself.  

Thanks & Regards,
Cheran



On 2/14/17, 9:32 PM, "Ken Giusti" <kgiu...@gmail.com> wrote:

>On Tue, Feb 14, 2017 at 2:52 PM, Elancheran Subramanian
><esubraman...@godaddy.com> wrote:
>> Hello All,
>> This is reg limiting the number of retries for Kafka driver support on 
>>Nova
>> and Neutron.
>>
>> While trying out the oslo messaging notifications support for Kafka on 
>>Nova
>> and Neutron, the Kafka driver doesn’t support limiting the number of 
>>retries
>> for failed messages. When I checked the code, currently there is no
>> configuration which support that, though the send_notification has retry
>> 
>>https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_d
>>rivers/impl_kafka.py#L336
>> but it’s not set or passed from component’s (nova or neutron’s)
>> configuration. Is there any configuration which I’m missing? Please let 
>>me
>> know.
>>
>
>You haven't missed anything - the kafka driver doesn't provide a means
>to set a default retry via its configuration.
>The expectation is that the caller (nova/neutron) would provide a
>retry value when constructing a Notifier instance.
>
>There was such a config option for the rabbitmq driver
>(rabbit_max_retries) but that was removed because it broke
>*something* - can't remember exactly the reason tho, sorry.
>
>>
>> Thanks in advance,
>> Cheran
>>
>> 
>>_
>>_
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: 
>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>
>-- 
>Ken Giusti  (kgiu...@gmail.com)
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo_messaging] Limiting the number of retries for kafka driver

[Elancheran Subramanian]

Hello All,
This is reg limiting the number of retries for Kafka driver support on Nova and 
Neutron.

While trying out the oslo messaging notifications support for Kafka on Nova and 
Neutron, the Kafka driver doesn’t support limiting the number of retries for 
failed messages. When I checked the code, currently there is no configuration 
which support that, though the send_notification has retry 
https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_kafka.py#L336
 but it’s not set or passed from component’s (nova or neutron’s) configuration. 
Is there any configuration which I’m missing? Please let me know.

Thanks in advance,
Cheran
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Should we drop kafka driver ?

[Elancheran Subramanian]

On 11/30/16, 4:45 PM, "Joshua Harlow"  wrote:


>Mehdi Abaakouk wrote:
>> Hi,
>>
>> I think my subject is clear :) , but I will add some facts that can help
>> to the decision:
>> * It uses only deprecated python-kafka API [1] [2]
>> * It's not python3 compatible [3]
>> * We still don't have kafka testing in gate
>> So, one year after the driver introduction, this one is still in bad 
>>shape
>> and doesn't match the requirements [4].
>>
>> These reviews looks abandoned/outdated/unmaintained:
>>
>> [1] https://review.openstack.org/#/c/297994/ [2]
>> https://review.openstack.org/#/c/332105/
>>
>> Other links:
>>
>> [3] https://review.openstack.org/#/c/404802/
>> [4]
>> 
>>http://docs.openstack.org/developer/oslo.messaging/supported-messaging-dr
>>ivers.html#testing
>>
>>
>> And of course, we will not drop the code now, but just deprecate it for
>> removal.
>> Cheers,
>>
>
>IMHO, not just yet, dims and I have been trying to use this driver 
>recently (for notifications only in my case) and I am more than willing 
>to try to get the changes needed to get this into a healthy state (from 
>my understanding dims and friends have been working through this as well).
>
>One of the places for gate testing that is still being worked on is the 
>following: https://github.com/jd/pifpaf/pull/28
>
>That will aid with some of the gate testing.
>
>-Josh

As Josh stated above, currently we have back ported the kafka driver to 
Stable/liberty https://github.com/tsecheran/oslo.messaging/pull/2 to get 
the notifications split work for Nova and Neutron in our cloud. It’s 
working fine and in early POC state, we need to still work on that pifpaf 
PR https://github.com/jd/pifpaf/pull/28 for the unit testing. 

- Cheran

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Splitting notifications from rpc (and questions + work around this)

[Elancheran Subramanian]

Hi Dims,
Thanks for sharing… 

Just wanted to check whether there is any development for Nova and Neutron 
going on, which we can leverage?  

Thanks,
Cheran




On 11/3/16, 12:51 AM, "Davanum Srinivas"  wrote:

>Josh,
>
>Kirill Bespalov put together this doc of which components will work
>with separate rpc and notification configurations:
>https://docs.google.com/document/d/1CU0KjL9iV8vut76hg9cFuWQGSJawuNq_cK7vRF
>_KyAA/edit?usp=sharing
>
>From my team, Oleksii Zamiatin is trying to scale up ZMQ beyond 200+
>nodes for RPC.
>
>Ilya Tyaptin's review is stuck because Monasca folks have trouble
>using the newer python-kafka version:
>https://review.openstack.org/#/c/332105/
>https://review.openstack.org/#/c/316259/
>
>As you can tell, we are trying to offer RabbitMQ or ZMQ for RPC and
>RabbitMQ or Kafka for Notifications.
>
>Hope this helps.
>
>Thanks,
>Dims
>
>On Wed, Nov 2, 2016 at 8:11 PM, Joshua Harlow  
>wrote:
>> Hi folks,
>>
>> There was a bunch of chatter at the summit about how there are really 
>>two
>> different types of (oslo) messaging usage that exist in openstack and 
>>how
>> they need not be backed by the same solution type (rabbitmq, qpid,
>> kafka...).
>>
>> For those that were not at the oslo sessions:
>>
>> https://wiki.openstack.org/wiki/Design_Summit/Ocata/Etherpads#Oslo
>>
>> The general gist was though that we need to make sure people really do 
>>know
>> that there are two very different types of messaging usage in openstack 
>>and
>> to ensure that operators (and developers) are picking the right backing
>> technology for each type.
>>
>> So some questions naturally arise out of this.
>>
>> * Where are the best practices with regard to selection of the best 
>>backend
>> type for rpc (and one for notifications); is this something 
>>oslo.messaging
>> should work through (or can the docs team and operator group also help 
>>in
>> making this)?
>>
>> * What are the tradeoffs in using the same (or different) technology 
>>for rpc
>> and notifications?
>>
>> * Is it even possible for all oslo.messaging consuming projects to be 
>>able
>> to choose 2 different backends, are consuming projects consuming the 
>>library
>> correctly so that they can use 2 different backends?
>>
>> * Is devstack able to run with say kafka for notifications and rabbitmq 
>>for
>> rpc (if not, is there any work item the oslo group can help with to make
>> this possible) so that we can ensure and test that all projects can work
>> correctly with appropriate (and possibly different) backends?
>>
>> * Any other messaging, arch-wg work that we (oslo or others) can help 
>>out
>> with to make sure that projects (and operators) are using the right
>> technology for the right use (and not just defaulting to RPC over 
>>rabbitmq
>> because it exists, when in reality something else might be a better 
>>choice)?
>>
>> * More(?)
>>
>> Just wanted to get this conversation started, because afaik it's one 
>>that
>> has not been widely circulated (and operators have been setting up 
>>rabbitmq
>> in various HA and clustered and ... modes, when in reality thinking 
>>through
>> what and how it is used may be more appropriate); this also applies to
>> developers since some technical solutions in openstack seem to be 
>>created
>> due to (in-part) rabbitmq shortcomings (cells v1 afaik was *in* part 
>>created
>> due to scaling issues).
>>
>> -Josh
>>
>> 
>>_
>>_
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: 
>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>-- 
>Davanum Srinivas :: https://twitter.com/dims
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA] Running Tempest tests for a customized cloud

[Elancheran Subramanian]

Hello Punal,
We do support both V2 and V3, that’s just a example I’ve stated BTW. We do have 
our own integration tests which are pretty much covers all our integration 
points with openstack. But we would like to leverage the tempest while doing 
our upstream merge for openstack components in CI.

I believe the tests support the include list, how can I exclude test? Any 
pointer would be a great help.

Thanks,
Cheran

From: punal patel <punal.pa...@gmail.com<mailto:punal.pa...@gmail.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Tuesday, August 16, 2016 at 4:16 PM
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [QA] Running Tempest tests for a customized cloud

Hi Cheran,

Best practice depends on your test plan and what coverage you. Does your test 
plan covers V3 Identity tests ? If it does and its failing, you should modify 
to make it work for your environment. Fast way to move forward is to exclude 
those tests.

-Punal

On Tue, Aug 16, 2016 at 3:40 PM, Elancheran Subramanian 
<esubraman...@godaddy.com<mailto:esubraman...@godaddy.com>> wrote:
Hello There,
I’m currently playing with using Tempest as our integration tests for our 
internal and external clouds, facing some issues with api which are not 
supported in our cloud. For ex, listing domains isn’t supported for any user, 
due to this V3 Identity tests are failing. So I would like to know what’s the 
best practice? Like fix those tests, and apply those fix as patch? Or just 
exclude those tests?

Would be great if anyone could share their experience on this.

Thanks & Regards,
Cheran

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [QA] Running Tempest tests for a customized cloud

[Elancheran Subramanian]

Hello There,
I’m currently playing with using Tempest as our integration tests for our 
internal and external clouds, facing some issues with api which are not 
supported in our cloud. For ex, listing domains isn’t supported for any user, 
due to this V3 Identity tests are failing. So I would like to know what’s the 
best practice? Like fix those tests, and apply those fix as patch? Or just 
exclude those tests?

Would be great if anyone could share their experience on this.

Thanks & Regards,
Cheran
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev