Re: [openstack-dev] [kolla] Mesos orchestration as discussed at mid cycle (action required from core reviewers)

[Geoff O'Callaghan]

I'm not a core dev but this is a fantastic idea which will have a huge
benefit to openstack.

Tks Geoff
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Outcome of distributed lock manager discussion @ the summit

[Geoff O'Callaghan]

On 03/11/2015 1:28 PM, "Robert Collins"  wrote:
>
> Hi, at the summit we had a big session on distributed lock managers
(DLMs).

Awesome.

>
> I'd just like to highlight the conclusions we came to in the session (
> https://etherpad.openstack.org/p/mitaka-cross-project-dlm
> )
>
> Firstly OpenStack projects that want to use a DLM can make it a hard
> dependency. Previously we've had a unwritten policy that DLMs should
> be optional, which has led to us writing poor DLM-like things backed
> by databases :(. So this is a huge and important step forward in our
> architecture.

Agreed and it's also a positive step.

>
> As in our existing pattern of usage for database and message-queues,
> we'll use an oslo abstraction layer: tooz. This doesn't preclude a
> different answer in special cases - but they should be considered
> special and exception, not the general case.
>
> Based on the project requirements surfaced in the discussion, it seems
> likely that all of konsul, etc and zookeeper will be able to have
> suitable production ready drivers written for tooz. Specifically no
> project required a fair locking implementation in the DLM.
>
> After our experience with oslo.messaging however, we wanted to avoid
> the situation of having unmaintained drivers and no signalling to
> users about them.
>
> So, we resolved to adopt roughly the oslo.messaging requirements for
> drivers, with a couple of tweaks...
>
> Production drivers in-tree will need:
>  - two nominated developers responsible for it
>  - gating functional tests that use dsvm
> Test drivers in-tree will need:
>  - clear identification that the driver is a test driver - in the
> module name at minimum
>
> All hail our new abstraction overlords.

This really is fantastic news.   Thanks for the 'heads up'.

Geoff

>
> -Rob
>
> --
> Robert Collins 
> Distinguished Technologist
> HP Converged Cloud
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][tc] Languages vs. Scope of "OpenStack"

[Geoff O'Callaghan]

Surely openstack is defined by it’s capabilities and interfaces rather than 
it’s internals.  Given the simplistic view that openstack is a collection of 
micro services connected by well defined api’s does it really matter what code 
is used inside that micro service (or macro service )?   If there is a 
community willing to bring and support code in language ‘x’,  isn’t that better 
than worrying about the off chance of existing developers wishing to move 
projects and not knowing the target language?Is there a fear that we’ll end 
up with a fork of nova (or others) written in rust ?
If we’re not open to evolving then we’ll die.

Just throwing in a different perspective.

Sorry about the top-post but it applies in general to the below discussion
Tks
Geoff

> On 24 May 2016, at 9:28 AM, Gregory Haynes  wrote:
> 
> On Mon, May 23, 2016, at 05:24 PM, Morgan Fainberg wrote:
>>  
>>  
>> On Mon, May 23, 2016 at 2:57 PM, Gregory Haynes > > wrote:
>> On Fri, May 20, 2016, at 07:48 AM, Thierry Carrez wrote:
>> > John Dickinson wrote:
>> > > [...]
>> > >> So the real question we need to answer is... where does OpenStack
>> > >> stop, and where does the wider open source community start ? If
>> > >> OpenStack is purely an "integration engine", glue code for other
>> > >> lower-level technologies like hypervisors, databases, or distributed
>> > >> block storage, then the scope is limited, Python should be plenty
>> > >> enough, and we don't need to fragment our community. If OpenStack is
>> > >> "whatever it takes to reach our mission", then yes we need to add one
>> > >> language to cover lower-level/native optimization, because we'll
>> > >> need that... and we need to accept the community cost as a
>> > >> consequence of that scope choice. Those are the only two options on
>> > >> the table.
>> > >>
>> > >> I'm actually not sure what is the best answer. But I'm convinced we,
>> > >> as a community, need to have a clear answer to that. We've been
>> > >> avoiding that clear answer until now, creating tension between the
>> > >> advocates of an ASF-like collection of tools and the advocates of a
>> > >> tighter-integrated "openstack" product. We have created silos and
>> > >> specialized areas as we got into the business of developing time-
>> > >> series databases or SDNs. As a result, it's not "one community"
>> > >> anymore. Should we further encourage that, or should we focus on
>> > >> what the core of our mission is, what we have in common, this
>> > >> integration engine that binds all those other open source projects
>> > >> into one programmable infrastructure solution ?
>> > >
>> > > You said the answer in your question. OpenStack isn't defined as an
>> > > integration engine[3]. The definition of OpenStack is whatever it
>> > > takes to fulfill our mission[4][5]. I don't mean that as a tautology.
>> > > I mean that we've already gone to the effort of defining OpenStack. It's
>> > > our mission statement. We're all about building a cloud platform upon
>> > > which people can run their apps ("cloud-native" or otherwise), so we
>> > > write the software needed to do that.
>> > >
>> > > So where does OpenStack stop and the wider community start? OpenStack
>> > > includes the projects needed to fulfill its mission.
>> >
>> > I'd totally agree with you if OpenStack was developed in a vacuum. But
>> > there is a large number of open source projects and libraries that
>> > OpenStack needs to fulfill its mission that are not in "OpenStack": they
>> > are external open source projects we depend on. Python, MySQL, libvirt,
>> > KVM, Ceph, OpenvSwitch, RabbitMQ... We are not asking that those should
>> > be included in OpenStack, and we are not NIHing replacements for those
>> > in OpenStack either.
>> >
>> > So it is not as clear-cut as you present it, and you can approach this
>> > dependency question from two directions.
>> >
>> > One is community-centric: "anything produced by our community is
>> > OpenStack". If we are missing a lower-level piece to achieve our mission
>> > and are developing it ourselves as a result, then it is OpenStack, even
>> > if it ends up being a message queue or a database.
>> >
>> > The other approach is product-centric: "lower-level pieces are OpenStack
>> > dependencies, rather than OpenStack itself". If we are missing a
>> > lower-level piece to achieve our mission and are developing it as a
>> > result, it could be developed on OpenStack infrastructure by members of
>> > the OpenStack community but it is not "OpenStack the product", it's an
>> > OpenStack *dependency*. It is not governed by the TC, it can use any
>> > language and tool deemed necessary.
>> >
>> > On this second approach, there is the obvious question of where
>> > "lower-level" starts, which as you explained above is not really
>> > clear-cut. A good litmus test for it could be whenever Python is not
>> > enough. If you can't develop it effectively with the language that is
>> > cu

Re: [openstack-dev] [all][tc] Languages vs. Scope of "OpenStack"

[Geoff O'Callaghan]

> On 24 May 2016, at 2:04 PM, Clint Byrum  wrote:
> 
> Excerpts from Geoff O'Callaghan's message of 2016-05-24 10:59:13 +1000:
>> Surely openstack is defined by it’s capabilities and interfaces rather than 
>> it’s internals.  Given the simplistic view that openstack is a collection of 
>> micro services connected by well defined api’s does it really matter what 
>> code is used inside that micro service (or macro service )?   If there is a 
>> community willing to bring and support code in language ‘x’,  isn’t that 
>> better than worrying about the off chance of existing developers wishing to 
>> move projects and not knowing the target language?Is there a fear that 
>> we’ll end up with a fork of nova (or others) written in rust ?
>> If we’re not open to evolving then we’ll die.
>> 
>> Just throwing in a different perspective.
> 
> Thanks Geoff. Your perspective is one that has been considered many
> times. That is an engineering perspective though, and ignores the people
> and businesses that are the users of OpenStack. We don't just shove the
> code out the door and say "good luck!”.

Hey Clint,  That is exactly what I wasn’t saying.   Businesses and people out 
there want the platform to have the features they want and work.  They could 
care less about what it’s written in.   You tend to care when it doesn’t work 
and / or it doesn’t have the features you want.   So I can understand for 
operators now they have a vested interest in making sure they can debug what is 
given to them as we don’t meet Geoff’s rule # 1 - the code must work and it 
must do what I want.

I also never said, ship the source code and say ‘good luck’.   What I did imply 
 was, due to a relaxing of coding platform requirements we might be able to 
deliver a function at this performance point that  we may not have been able to 
do otherwise.   We should always provide support and the code,  but as to what 
language it’s written it i’m personally not fussed and I have to deal with a 
variety of languages already so maybe that’s why I don’t see it as a big 
problem.

I understand there will be integration challenges and I agree with cohesiveness 
being a good thing, but I also believe we must deliver value more than 
cohesiveness.   The value is what operators want,  the cohesiveness is what the 
developers may or may not want.

> 
> So, each new aspect of the internals that the people and business must
> evaluate is another hurdle to adoption at some level. If OpenStack is
> the fastest open source cloud platform ever, but only gets adopted by
> the 5 largest cloud users on the planet, and smaller organizations are
> forced to invest their money in closed source appliance based clouds,
> then we'll all suffer, as neither of those options will stand a chance
> against the large scale efforts of the established players like Amazon,
> Google, and Microsoft.
> 
> Likewise, if OpenStack is adopted by the bulk of small/medium businesses
> needing clouds, but nobody can run it at scale, we will also be crushed
> as users develop elastic workloads that need to be moved onto a large
> scale cloud.
> 
> And if there are simply two, one for big clouds, and one for small,
> they'll never be fully compatible. It will be like POSIX... moderately
> useful for a limited set of applications, but most things will have to
> be optimized for each different implementation, wasting everyone's time
> porting when they should be developing.
> 
> So, while this doesn't mean we should just force everything onto Python,
> it does mean that we should remain as cohesive as we can when making
> choices like this. So the question "What is OpenStack" needs to be
> asked, so we can evaluate what should be kept close together, and what
> might be better off as an independent component.

I thought that had already been asked and answered.  The question becomes :- is 
what openstack is written in a core part of openstack?

Thanks for the reply.
Geoff


> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][tc] Languages vs. Scope of "OpenStack"

[Geoff O'Callaghan]

> On 24 May 2016, at 3:13 PM, Clint Byrum  wrote:
> 
> 
[snip]

> those other needs. Grab a python developer, land some code, and your
> feature is there.

s/python/whateverlanguage/

> 
>> I also never said, ship the source code and say ‘good luck’.   What I did 
>> imply  was, due to a relaxing of coding platform requirements we might be 
>> able to deliver a function at this performance point that  we may not have 
>> been able to do otherwise.   We should always provide support and the code,  
>> but as to what language it’s written it i’m personally not fussed and I have 
>> to deal with a variety of languages already so maybe that’s why I don’t see 
>> it as a big problem.
> 
> This again assumes that one only buys software and does not ever
> participate in its development in an ongoing basis. There's nothing
> wrong with that, but this particular community is highly focused on
> people who do want to participate and think that the ability to
> adapt this cloud they've invested in to their changing business needs is
> more important than any one feature.

No I didn’t say that at all and I don’t believe it’s assumed.I just said I 
wasn’t fussed about what language it’s written in and just wanted developers to 
be able to contribute if they had something to contribute.   

> 
>> 
>> I understand there will be integration challenges and I agree with 
>> cohesiveness being a good thing, but I also believe we must deliver value 
>> more than cohesiveness.   The value is what operators want,  the 
>> cohesiveness is what the developers may or may not want.
>> 
> 
> We agree that delivering value to end users and operators is the #1
> priority. I think we just disagree on the value of an open development
> model and cohesion in the _community_.

It’s not open if you restrict developers based on programming language.
Trust me I get cohesion and it’s value, we’ve reached the stage now where 
cohesion is being questioned.  The questioning is a good thing and it is a 
measure of the health of the community.

> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Oslo messaging vs zaqar

[Geoff O'Callaghan]

Hi all,
I'm just trying to understand the messaging strategy in openstack.It
seems we have at least 2 messaging layers.

Oslo.messaging and zaqar,  Can someone explain to me why there are two?
To quote from the zaqar faq :
-
How does Zaqar compare to oslo.messaging?

oslo.messsaging is an RPC library used throughout OpenStack to manage
distributed commands by sending messages through different messaging
layers. Oslo Messaging was originally developed as an abstraction over
AMQP, but has since added support for ZeroMQ.

As opposed to Oslo Messaging, Zaqar is a messaging service for the over and
under cloud. As a service, it is meant to be consumed by using libraries
for different languages. Zaqar currently supports 1 protocol (HTTP) and
sits on top of other existing technologies (MongoDB as of version 1.0).

It seems to my casual view that we could have one and scale that and use it
for SQS style messages, internal messaging (which could include logging)
all managed by message schemas and QoS.  This would give a very robust and
flexible system for endpoints to consume.

Is there a plan to consolidate?

Rgds
Geoff
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Oslo messaging vs zaqar

[Geoff O'Callaghan]

So
On 22/09/2014 10:01 PM, "Zane Bitter"  wrote:
>
> On 20/09/14 04:17, Geoff O'Callaghan wrote:
>>
>> Hi all,
>> I'm just trying to understand the messaging strategy in openstack.It
>> seems we have at least 2 messaging layers.
>>
>> Oslo.messaging and zaqar,  Can someone explain to me why there are two?
>>
>> Is there a plan to consolidate?
>
>
> I'm trying to understand the database strategy in OpenStack. It seems we
have at least 2 database layers - sqlalchemy and Trove.
>
> Can anyone explain to me why there are two?
>
>
> Is there a plan to consolidate?
> 
>

So the answer is because we can ;)  Not a great answer, but an answer
nonetheless.  :)

That being said I'm not sure why a well constructed zaqar with an rpc
interface couldn't meet the requirements of oslo.messsaging and much
more.It seems I need to dig some more.

Thanks all for taking the time to reply.

Geoff
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Oslo messaging vs zaqar

[Geoff O'Callaghan]

On 23/09/2014 1:59 AM, "Clint Byrum"  wrote:
>
> Geoff, do you expect all of our users to write all of their messaging
> code in Python?
>
> oslo.messaging is a _python_ library.
>
> Zaqar is a service with a REST API -- accessible to any application.

No I do not.   I am suggesting thaf a well designed, scalable and robust
messaging layer can meet the requirements of both as well as a number of
other openstack servuces.  How the messaging layer is consumed isn't the
issue.

Below is what I originally posted.

> > 
> > It seems to my casual view that we could have one and scale that and
use it
> > for SQS style messages, internal messaging (which could include logging)
> > all managed by message schemas and QoS.  This would give a very robust
and
> > flexible system for endpoints to consume.
> >
> > Is there a plan to consolidate?

Tks
Geoff
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Oslo messaging vs zaqar

[Geoff O'Callaghan]

On 23/09/2014 12:58 PM, "Clint Byrum"  wrote:
>
> Excerpts from Geoff O'Callaghan's message of 2014-09-22 17:30:47 -0700:
> > On 23/09/2014 1:59 AM, "Clint Byrum"  wrote:
> > >
> > > Geoff, do you expect all of our users to write all of their messaging
> > > code in Python?
> > >
> > > oslo.messaging is a _python_ library.
> > >
> > > Zaqar is a service with a REST API -- accessible to any application.
> >
> > No I do not.   I am suggesting thaf a well designed, scalable and robust
> > messaging layer can meet the requirements of both as well as a number of
> > other openstack servuces.  How the messaging layer is consumed isn't the
> > issue.
> >
> > Below is what I originally posted.
> >
> > > > 
> > > > It seems to my casual view that we could have one and scale that and
> > use it
> > > > for SQS style messages, internal messaging (which could include
logging)
> > > > all managed by message schemas and QoS.  This would give a very
robust
> > and
> > > > flexible system for endpoints to consume.
> > > >
> > > > Is there a plan to consolidate?
> >
>
> Sorry for the snark George. I was very confused by the text above, and
> I still am. I am confused because consolidation requires commonalities,
> of which to my mind, there are almost none other than the relationship
> to the very abstract term "messaging".

Hahaha.. now you're calling me george ;)   Don't worry dude.  I'm only
joking and I even liked sharknado.   Any way I was hoping zaqar had a
greater scope than it appears to have.   I'll watch the progress.

>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Glance] IRC logging

[Geoff O'Callaghan]

On 13/01/2015 3:45 AM, "Jeremy Stanley"  wrote:
>
> There's really no way to _force_ official logging on all
> project-related channels. People who are opposed to the idea simply
> move their conversations to new channels. They'll straddle the line
> between "somewhat official looking" and "official enough to require
> logging."

Yeah while this is true you can certainly work to achieve this goal and if
there are side conversations that aren't logged then so be it.

Irc logs can be a very valuable resource.

Thanks
Geoff
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Scheduler proposal

[Geoff O'Callaghan]

On 11/10/2015 6:25 PM, "Clint Byrum"  wrote:
>
> Excerpts from Boris Pavlovic's message of 2015-10-11 00:02:39 -0700:
> > 2Everybody,
> >
> > Just curios why we need such complexity.
> >
> >
> > Let's take a look from other side:
> > 1) Information about all hosts (even in case of 100k hosts) will be less
> > then 1 GB
> > 2) Usually servers that runs scheduler service have at least 64GB RAM
and
> > more on the board
> > 3) math.log(10) < 12  (binary search per rule)
> > 4) We have less then 20 rules for scheduling
> > 5) Information about hosts is updated every 60 seconds (no updates host
is
> > dead)

[Snip]

>
> I'm in, except I think this gets simpler with an intermediary service
> like ZK/Consul to keep track of this 1GB of data and replace the need
> for 6, and changes the implementation of 5 to "updates its record and
> signals its presence".

I have to agree,  something like ZK looks like it'd make things simpler and
they're in general well proven technology  (esp ZK)
They handle the centralized coordination well and all the hard resiliency
is thrown in.

Geoff
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev