Re: [openstack-dev] [Trove] Core reviewer update

[Tim Simpson]

+1

Alas, my days of offering substantial activity on Trove are over. Best of luck 
in all of your adventures!



On Feb 5, 2015, at 10:26 AM, Nikhil Manchanda 
mailto:slick...@gmail.com>>
 wrote:

Hello Trove folks:

Keeping in line with other OpenStack projects, and attempting to keep
the momentum of reviews in Trove going, we need to keep our core-team up
to date -- folks who are regularly doing good reviews on the code should
be brought in to core and folks whose involvement is dropping off should
be considered for removal since they lose context over time, not being
as involved.

For this update I'm proposing the following changes:
- Adding Peter Stachowski (peterstac) to trove-core
- Adding Victoria Martinez De La Cruz (vkmc) to trove-core
- Adding Edmond Kotowski (edmondk) to trove-core
- Removing Michael Basnight (hub_cap) from trove-core
- Removing Tim Simpson (grapex) from trove-core

For context on Trove reviews and who has been active, please see
Russell's stats for Trove at:
- http://russellbryant.net/openstack-stats/trove-reviewers-30.txt
- http://russellbryant.net/openstack-stats/trove-reviewers-90.txt

Trove-core members -- please reply with your vote on each of these
proposed changes to the core team. Peter, Victoria and Eddie -- please
let me know of your willingness to be in trove-core. Michael, and Tim --
if you are planning on being substantially active on Trove in the near
term, also please do let me know.

Thanks,
Nikhil
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org<mailto:openstack-dev-requ...@lists.openstack.org>?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Trove] Guest RPC API improvements. Messages, topics, queues, consumption.

[Tim Simpson]

Hi Denis,

It seems like the issue you're trying to solve is that these 'prepare' messages 
can't be consumed by the guest.
So, if the guest never actually comes online and therefore can't consume the 
prepare call, then you'll be left with the message in the queue forever.

If you use a ping-pong message, you'll still be left with a stray message in 
the queue if it fails.

I think the best fix is if we delete the queue when deleting an instance. This 
way you'll never have more queues in rabbit than are needed.

Thanks,

Tim



From: Denis Makogon [dmako...@mirantis.com]
Sent: Friday, October 31, 2014 4:32 AM
To: OpenStack Development Mailing List
Subject: [openstack-dev] [Trove] Guest RPC API improvements. Messages, topics, 
queues, consumption.


Hello, Stackers/Trovers.



I’d like to start discussion about how do we use guestagent API that will 
eventually be evaluated as a spec. For most of you who well-known with Trove’s 
codebase knows how do Trove acts when provisioning new instance.

I’d like to point out next moments:

  1.  When we provision new instance we expect that guest will create its 
topic/queue for RPC messaging needs.

  2.  Taskmanager doesn’t validate that guest is really up before sending 
‘prepare’ call.

And here comes the problem, what if guest wasn’t able to start properly and 
consume ‘prepare’ message due to certain circumstances? In this case ‘prepare’ 
message would never be consumed.


Me and Sergey Gotliv were looking for proper solution for this case. And we end 
up with next requirements for provisioning workflow:

  1.  We must be sure that ‘prepare’ message will be consumed by guest.

  2.  Taskmanager should handle topic/queue management for guest.

  3.  Guest just need to consume income messages for already existing 
topic/queue.

As concrete proposal (or at least topic for discussions) i’d like to discuss 
next improvements:

We need to add new guest RPC API that will represent “ping-pong” action. So 
before sending any cast- or call-type messages we need to make sure that guest 
is really running.


Pros/Cons for such solution:

  1.  Guest will do only consuming.

  2.  Guest would not manage its topics/queues.

  3.  We’ll be 100% sure that no messages would be lost.

  4.  Fast-fail during provisioning.

  5.  Other minor/major improvements.



Thoughts?


P.S.: I’d like to discuss this topic during upcoming Paris summit (during 
contribution meetup at Friday).



Best regards,

Denis Makogon

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Trove] Proposal to add Iccha Sethi to trove-core

[Tim Simpson]

+1 


From: Nikhil Manchanda [nik...@manchanda.me]
Sent: Thursday, October 30, 2014 3:47 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [Trove] Proposal to add Iccha Sethi to trove-core

Hello folks:

I'm proposing to add Iccha Sethi (iccha on IRC) to trove-core.

Iccha has been working with Trove for a while now. She has been a
very active reviewer, and has provided insightful comments on
numerous reviews. She has submitted quality code for multiple bug-fixes
in Trove, and most recently drove the per datastore volume support BP in
Juno. She was also a crucial part of the team that implemented
replication in Juno, and helped close out multiple replication related
issues during Juno-3.

https://review.openstack.org/#/q/reviewer:"iccha",n,z
https://review.openstack.org/#/q/owner:"iccha",n,z

Please respond with +1/-1, or any further comments.

Thanks,
Nikhil

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Trove] Cluster implementation is grabbing instance's gutsHi guys, I was looking through the clustering code today and noticed a lot of it is grabbing what I'd call the guts of the ins

[Tim Simpson]

Hi everyone,

I was looking through the clustering code today and noticed a lot of it is 
grabbing what I'd call the guts of the instance models code.

The best example is here: 
https://github.com/openstack/trove/commit/06196fcf67b27f0308381da192da5cc8ae65b157#diff-a4d09d28bd2b650c2327f5d8d81be3a9R89https://github.com/openstack/trove/commit/06196fcf67b27f0308381da192da5cc8ae65b157#diff-a4d09d28bd2b650c2327f5d8d81be3a9R89

In the "_all_instances_ready" function, I would have expected 
trove.instance.models.load_any_instance to be called for each instance ID and 
it's status to be checked.

Instead, the service_status is being called directly. That is a big mistake. 
For now it works, but in general it mixes the concern of "what is an instance 
stauts?" to code outside of the instance class itself.

For an example of why this is bad, look at the method 
"_instance_ids_with_failures." The code is checking for failures by seeing if 
the service status is failed. What if the Nova server or Cinder volume have 
tanked instead? The code won't work as expected.

It could be we need to introduce another status besides BUILD to instance 
statuses, or we need to introduce a new internal property to the SimpleInstance 
base class we can check. But whatever we do we should add this extra logic to 
the instance class itself rather than put it in the clustering models code.

This is a minor nitpick but I think we should fix it before too much time 
passes.

Thanks,

Tim
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Trove] Proposal to add Amrith Kumar to trove-core

[Tim Simpson]

+1


From: Sergey Gotliv [sgot...@redhat.com]
Sent: Tuesday, August 26, 2014 8:11 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Trove] Proposal to add Amrith Kumar to trove-core

Strong +1 from me!


> -Original Message-
> From: Nikhil Manchanda [mailto:nik...@manchanda.me]
> Sent: August-26-14 3:48 AM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: [openstack-dev] [Trove] Proposal to add Amrith Kumar to trove-core
>
> Hello folks:
>
> I'm proposing to add Amrith Kumar (amrith on IRC) to trove-core.
>
> Amrith has been working with Trove for a while now. He has been a
> consistently active reviewer, and has provided insightful comments on
> numerous reviews. He has submitted quality code for multiple bug-fixes in
> Trove, and most recently drove the audit and clean-up of log messages across
> all Trove components.
>
> https://review.openstack.org/#/q/reviewer:"amrith",n,z
> https://review.openstack.org/#/q/owner:"amrith",n,z
>
> Please respond with +1/-1, or any further comments.
>
> Thanks,
> Nikhil
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Glance][Trove] Metadata Catalog

[Tim Simpson]

I agree as well.

I think we should spend less time worrying about what other projects in 
OpenStack might do in the future and spend more time on adding the features we 
need today to Trove. I understand that it's better to work together but too 
often we stop progress on something in Trove to wait on a feature in another 
project that is either incomplete or merely being planned.

While this stems from our strong desire to be part of the community, which is a 
good thing, it hasn't actually led many of us to do work for these other 
projects. At the same time, its negatively impacted Trove. I also think it 
leads us to over-design or incorrectly design features as we plan for 
functionality in other projects that may never materialize in the forms we 
expect.

So my vote is we merge our own metadata feature and not fret over how metadata 
may end up working in Glance.

Thanks,

Tim


From: Iccha Sethi [iccha.se...@rackspace.com]
Sent: Thursday, July 24, 2014 4:02 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Glance][Trove] Metadata Catalog

+1

We are unsure when these changes will get into glance.
IMO we should go ahead will our instance metadata patch for now and when things 
are ready in glance land we can consider migrating to using that as a generic 
metadata repository.

Thanks,
Iccha

From: Craig Vyvial mailto:cp16...@gmail.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
mailto:openstack-dev@lists.openstack.org>>
Date: Thursday, July 24, 2014 at 3:04 PM
To: "OpenStack Development Mailing List (not for usage questions)" 
mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [Glance][Trove] Metadata Catalog

Denis,

The scope of the metadata api goes beyond just using the glance metadata. The 
metadata can be used for instances and and other objects to add extra data like 
tags or something else that maybe a UI might want to use. We need this feature 
either way.

-Craig


On Thu, Jul 24, 2014 at 12:17 PM, Amrith Kumar 
mailto:amr...@tesora.com>> wrote:
Speaking as a ‘database guy’ and a ‘Trove guy’, I’ll say this; “Metadata” is a 
very generic term and the meaning of “metadata” in a database context is very 
different from the meaning of “metadata” in the context that Glance is 
providing.

Furthermore the usage and access pattern for this metadata, the frequency of 
change, and above all the frequency of access are fundamentally different 
between Trove and what Glance appears to be offering, and we should probably 
not get too caught up in the project “title”.

We would not be “reinventing the wheel” if we implemented an independent 
metadata scheme for Trove; we would be implementing the right kind of when for 
the vehicle that we are operating. Therefore I do not agree with your 
characterization that concludes that:

>> given goals at [1] are out of scope of Database program, etc

Just to be clear, when you write:

>> Unfortunately, we’re(Trove devs) are on half way to metadata …

it is vital to understand that our view of “metadata” is very different from 
(for example, a file system’s view of metadata, or potentially Glance’s view of 
metadata). For that reason, I believe that your comments on 
https://review.openstack.org/#/c/82123/16 are also somewhat extreme.

Before postulating a solution (or “delegating development to Glance devs”), it 
would be more useful to fully describe the problem being solved by Glance and 
the problem(s) we are looking to solve in Trove, and then we could have a 
meaningful discussion about the right solution.

I submit to you that we will come away concluding that there is a round peg, 
and a square hole. Yes, one will fit in the other but the final product will 
leave neither party particularly happy with the end result.

-amrith

From: Denis Makogon [mailto:dmako...@mirantis.com]
Sent: Thursday, July 24, 2014 9:33 AM
To: OpenStack Development Mailing List
Subject: [openstack-dev] [Glance][Trove] Metadata Catalog


Hello, Stackers.

 I’d like to discuss the future of Trove metadata API. But first small 
history info (mostly taken for Trove medata spec, see [1]):
Instance metadata is a feature that has been requested frequently by our users. 
They need a way to store critical information for their instances and have that 
be associated with the instance so that it is displayed whenever that instance 
is listed via the API. This also becomes very usable from a testing perspective 
when doing integration/ci. We can utilize the metadata to store things like 
what process created the instance, what the instance is being used for, etc... 
The design for this feature is modeled heavily on the Nova metadata API with a 
few tweaks in how it works internally.

And here comes conflict. Glance devs are working on “Glance Metadata 
Catalog” feature (see [2]). And as for me, we don’t have to “reinvent the

Re: [openstack-dev] [Trove] Guest prepare call polling mechanism issue

[Tim Simpson]

To summarize, this is a conversation about the following LaunchPad bug: 
https://launchpad.net/bugs/1325512
and Gerrit review: https://review.openstack.org/#/c/97194/6

You are saying the function "_service_is_active" in addition to polling the 
datastore service status also polls the status of the Nova resource. At first I 
thought this wasn't the case, however looking at your pull request I was 
surprised to see on line 320 
(https://review.openstack.org/#/c/97194/6/trove/taskmanager/models.py) polls 
Nova using the "get" method (which I wish was called "refresh" as to me it 
sounds like a lazy-loader or something despite making a full GET request each 
time).
So moving this polling out of there into the two respective "create_server" 
methods as you have done is not only going to be useful for Heat and avoid the 
issue of calling Nova 99 times you describe but it will actually help 
operations teams to see more clearly that the issue was with a server that 
didn't provision. We actually had an issue in Staging the other day that took 
us forever to figure out because the server wasn't provisioning, but before 
anything checked that it was ACTIVE the DNS code detected the server had no ip 
address (never mind it was in a FAILED state) so the logs surfaced this as a 
DNS error. This change should help us avoid such issues.

Thanks,

Tim



From: Denis Makogon [dmako...@mirantis.com]
Sent: Wednesday, July 23, 2014 7:30 AM
To: OpenStack Development Mailing List
Subject: [openstack-dev] [Trove] Guest prepare call polling mechanism issue


Hello, Stackers.


I’d like to discuss guestagent prepare call polling mechanism issue (see [1]).


Let me first describe why this is actually an issue and why it should be fixed. 
For those of you who is familiar with Trove knows that Trove can provision 
instances through Nova API and Heat API (see [2] and see [3]).



What’s the difference between this two ways (in general)? The answer is 
simple:

- Heat-based provisioning method has polling mechanism that verifies that stack 
provisioning was completed with successful state (see [4]) which means that all 
stack resources are in ACTIVE state.

- Nova-based provisioning method doesn’t do any polling (which is wrong, since 
instance can’t fail as fast as possible because Trove-taskmanager service 
doesn’t verify that launched server had reached ACTIVE state. That’s the issue 
#1 - compute instance state is unknown, but right after resources (deliverd by 
heat) already in ACTIVE states.


Once one method [2] or [3] finished, taskmanager trying to prepare data for 
guest (see [5]) and then it tries to send prepare call to guest (see [6]). Here 
comes issue #2 - polling mechanism does at least 100 API calls to Nova to 
define compute instance status.

Also taskmanager does almost the same amount of calls to Trove backend to 
discover guest status which is totally normal.


So, here comes the question,  why should i call 99 times Nova for the same 
value if the value asked for the first time was completely acceptable?



There’s only one way to fix it. Since heat-based provisioning delivers 
instance with status validation procedure, the same thing should be done for 
nova-base provisioning (we should extract compute instance status polling from 
guest prepare polling mechanism and integrate it into [2]) and leave only guest 
status discovering in guest prepare polling mechanism.





Benefits? Proposed fix will give an ability for fast-failing for corrupted 
instances, it would reduce amount of redundant Nova API calls while attempting 
to discover guest status.



Proposed fix for this issue - [7].


[1] - https://launchpad.net/bugs/1325512

[2] - 
https://github.com/openstack/trove/blob/master/trove/taskmanager/models.py#L198-L215

[3] - 
https://github.com/openstack/trove/blob/master/trove/taskmanager/models.py#L190-L197

[4] - 
https://github.com/openstack/trove/blob/master/trove/taskmanager/models.py#L420-L429

[5] - 
https://github.com/openstack/trove/blob/master/trove/taskmanager/models.py#L217-L256

[6] - 
https://github.com/openstack/trove/blob/master/trove/taskmanager/models.py#L254-L266

[7] - https://review.openstack.org/#/c/97194/



Thoughts?


Best regards,

Denis Makogon
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [trove] guestagent config for overriding managers

[Tim Simpson]

I’m a fan of the later suggestion.
Thanks,

Tim

From: Craig Vyvial [mailto:cp16...@gmail.com]
Sent: Tuesday, July 01, 2014 11:35 PM
To: OpenStack Development Mailing List
Subject: [openstack-dev] [trove] guestagent config for overriding managers

If you want to override the trove guestagent managers its looks really nasty to 
have EVERY manager on a single line here.

datastore_registry_ext = 
mysql:my.guestagent.datastore.mysql.manager.Manager,percona:my.guestagent.datastore.mysql.manager.Manager,...

This needs to be tidied up and split out some way.
Ideally each of these should be on a single line.

datastore_registry_ext = mysql:my.guestagent.datastore.mysql.manager.Manager
datastore_registry_ext = percona:my.guestagent.datastore.mysql.manager.Manager

or maybe...

datastores = mysql,precona
[mysql]
manager = my.guestagent.datastore.mysql.manager.Manager
[percona]
manager = my.guestagent.datastore.percona.manager.Manager

After typing out the second idea i dont like it as much as something like the 
first way.

Thoughts?

Thanks,
- Craig Vyvial
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Trove] Proposal to add Craig Vyvial to trove-core

[Tim Simpson]

+1

From: Peter Stachowski [pe...@tesora.com]
Sent: Tuesday, May 06, 2014 9:06 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Trove] Proposal to add Craig Vyvial to trove-core

+1

-Original Message-
From: Nikhil Manchanda [mailto:nik...@manchanda.me]
Sent: May-06-14 5:32 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [Trove] Proposal to add Craig Vyvial to trove-core


Hello folks:

I'm proposing to add Craig Vyvial (cp16net) to trove-core.

Craig has been working with Trove for a while now. He has been a consistently 
active reviewer, and has provided insightful comments on numerous reviews. He 
has submitted quality code to multiple features in Trove, and most recently 
drove the implementation of configuration groups in Icehouse.

https://review.openstack.org/#/q/reviewer:%22Craig+Vyvial%22,n,z
https://review.openstack.org/#/q/owner:%22Craig+Vyvial%22,n,z

Please respond with +1/-1, or any further comments.

Thanks,
Nikhil

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [trove] scheduled tasks redux

[Tim Simpson]

>>  Would it make more sense for an operator to configure a "time window", and 
>> then let users choose a slot within a time window (and say there are a 
>> finite number of slots in a time window). The slotting would be done behind 
>> the scenes and a user would only be able to select a window, and if the 
>> slots are all taken, it wont be shown in the "get available time windows". 
>> the "available time windows" could be smart, in that, your avail time 
>> window _could be_ based on the location of the hardware your vm is sitting 
>> on (or some other rule…). Think network saturation if everyone on host A is 
>> doing a backup to swift.

Allowing operators to define time windows seems preferable to me; I think a 
cron like system might be too granular. Having windows seems easier to schedule 
and would enable an operator to change things in a pinch.

From: Michael Basnight [mbasni...@gmail.com]
Sent: Thursday, January 23, 2014 3:41 PM
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] [trove] scheduled tasks redux

On Jan 23, 2014, at 12:20 PM, Greg Hill wrote:

> The blueprint is here:
>
> https://wiki.openstack.org/wiki/Trove/scheduled-tasks
>
> So I have basically two questions:
>
> 1. Does anyone see a problem with defining the repeating options as a single 
> field rather than multiple fields?

Im fine w/ a single field, but more explanation below.

> 2. Should we use the crontab format for this or is that too terse?  We could 
> go with a more fluid style like "Every Wednesday/Friday/Sunday at 12:00PM" 
> but that's English-centric and much more difficult to parse programmatically. 
>  I'd welcome alternate suggestions.

Will we be doing more complex things than "every day at some time"? ie, does 
the user base see value in configuring backups every 12th day of every other 
month? I think this is easy to write the schedule code, but i fear that it will 
be hard to build a smarter scheduler that would only allow X tasks in a given 
hour for a window. If we limit to daily at X time, it seems easier to estimate 
how a given window for backup will look for now and into the future given a 
constant user base :P Plz note, I think its viable to schedule more than 1 per 
day, in cron "* 0,12" or "* */12".

Will we be using this as a single task service as well? So if we assume the 
first paragraph is true, that tasks are scheduled daily, single task services 
would be scheduled once, and could use the same crontab fields. But at this 
point, we only really care about the minute, hour, and _frequency_, which is 
daily or once. Feel free to add 12 scheduled tasks for every 2 hours if you 
want to back it up that often, or a single task as * 0/2. From the backend, i 
see that as 12 tasks created, one for each 2 hours.

But this doesnt take into mind windows, when you say you want a cron style 2pm 
backup, thats really just during some available window. Would it make more 
sense for an operator to configure a "time window", and then let users choose a 
slot within a time window (and say there are a finite number of slots in a time 
window). The slotting would be done behind the scenes and a user would only be 
able to select a window, and if the slots are all taken, it wont be shown in 
the "get available time windows". the "available time windows" could be smart, 
in that, your avail time window _could be_ based on the location of the 
hardware your vm is sitting on (or some other rule…). Think network saturation 
if everyone on host A is doing a backup to swift.

/me puts down wrench

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [trove] Dropping connectivity from guesagent to Trove back-end

[Tim Simpson]

>> whose proposed future phases include turning conductor into a source of 
>> truth for trove to ask about instances, and then using its own datastore 
>> separate from the host db anyway.

IIRC this was to support such ideas as storing the heart beat or service status 
somewhere besides the Trove database. So let's say that instead of having to 
constantly update the heart beat table from the guest it was possible to ask 
Rabbit when the last time the guest tried to receive a message and use that as 
the heartbeat timestamp instead. This is what Conductor was meant to support - 
the ability to not force a guest to have to send back heart beat info to a 
database if there was an RPC technology dependent way to get that info which 
Conductor knew about.

I don't agree with the idea that all information on a guest should live only in 
Conductor. Under this logic we'd have no backup information in the Trove 
database we could use when listing backups and would have to call Conductor 
instead.  I don't see what that buys us.

Similarly with the RootHistory object, it lives in the database right now which 
works fine because anytime Root is enabled it's done by Trove code which has 
access to that database anyway. Moving root history to Conductor will 
complicate things without giving us any benefit.

Thanks,

Tim


From: Ed Cranford [ed.cranf...@gmail.com]
Sent: Friday, December 20, 2013 10:13 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [trove] Dropping connectivity from guesagent to 
Trove back-end

Conductor was the first phase of 
https://wiki.openstack.org/wiki/Trove/guest_agent_communication whose proposed 
future phases include turning conductor into a source of truth for trove to ask 
about instances, and then using its own datastore separate from the host db 
anyway.
The purpose of the root history table is to keep information in a place even an 
instance with root cannot reach, so we essentially have a warranty seal on the 
instance. The thinking at was if that status was kept on the instance, intrepid 
users could potentially enable root, muck about, and then manually remove root. 
By putting that row in a table outside the instance there's no question.

Phase 2 of the document above is to make conductor the source of truth for 
information about an instance, so taskman will start asking conductor instead 
of fetching the database information directly. So I think the next step for 
removing this is to give conductor a method taskman can call to get the root 
status from the extant table.

Phase 3 then seeks to give conductor its own datastore away from the original 
database; I think that's the right time to migrate the root history table, too.


On Fri, Dec 20, 2013 at 9:44 AM, Denis Makogon 
mailto:dmako...@mirantis.com>> wrote:
Unfortunately, Trove cannot manage it's own extensions, so if, suppose, i would 
try to get provisioned cassandra instance i would be still possible to check if 
root enabled.
Prof: 
https://github.com/openstack/trove/blob/master/trove/extensions/mysql/service.py
There are no checks for datastore_type, service just loads root model and 
that's it, since my patch create root model, next API call (root check) will 
load this model.



2013/12/20 Tim Simpson 
mailto:tim.simp...@rackspace.com>>
Because the ability to check if root is enabled is in an extension which would 
not be in effect for a datastore with no ACL support, the user would not be 
able to see that the marker for root enabled was set in the Trove 
infrastructure database either way.

By the way- I double checked the code, and I was wrong- the guest agent was 
*not* telling the database to update the root enabled flag. Instead, the API 
extension had been updating the database all along after contacting the guest. 
Sorry for making this thread more confusing.

It seems like if you follow my one (hopefully last) suggestion on this pull 
request, it will solve the issue you're tackling: 
https://review.openstack.org/#/c/59410/5

Thanks,

Tim


From: Denis Makogon [dmako...@mirantis.com<mailto:dmako...@mirantis.com>]
Sent: Friday, December 20, 2013 8:58 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [trove] Dropping connectivity from guesagent to 
Trove back-end

Thanks for response, Tim.

As i said, it would be confusing situation when database which has no ACL would 
be deployed by Trove with root enabled - this looks very strange since user 
allowed to check if root enabled. I think in this case Conductor should be 
_that_ place which should contain datastore specific logic, which requires 
back-end connectivity.

It would be nice to have consistent instance states for each datastore types 
and version.

Are there any objections about letting conductor d

Re: [openstack-dev] [trove] Dropping connectivity from guesagent to Trove back-end

[Tim Simpson]

I think you're addressing a different problem which is that the extensions for 
MySQL shouldn't always apply to every single datastore. However I think we 
should proceed on the assumption that this will be fixed.

Btw, last time we tried to address it there was a week of awful, three hour 
meetings and we couldn't reach consensus.

Thanks,

Tim

From: Denis Makogon [dmako...@mirantis.com]
Sent: Friday, December 20, 2013 9:44 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [trove] Dropping connectivity from guesagent to 
Trove back-end

Unfortunately, Trove cannot manage it's own extensions, so if, suppose, i would 
try to get provisioned cassandra instance i would be still possible to check if 
root enabled.
Prof: 
https://github.com/openstack/trove/blob/master/trove/extensions/mysql/service.py
There are no checks for datastore_type, service just loads root model and 
that's it, since my patch create root model, next API call (root check) will 
load this model.


2013/12/20 Tim Simpson 
mailto:tim.simp...@rackspace.com>>
Because the ability to check if root is enabled is in an extension which would 
not be in effect for a datastore with no ACL support, the user would not be 
able to see that the marker for root enabled was set in the Trove 
infrastructure database either way.

By the way- I double checked the code, and I was wrong- the guest agent was 
*not* telling the database to update the root enabled flag. Instead, the API 
extension had been updating the database all along after contacting the guest. 
Sorry for making this thread more confusing.

It seems like if you follow my one (hopefully last) suggestion on this pull 
request, it will solve the issue you're tackling: 
https://review.openstack.org/#/c/59410/5

Thanks,

Tim


From: Denis Makogon [dmako...@mirantis.com<mailto:dmako...@mirantis.com>]
Sent: Friday, December 20, 2013 8:58 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [trove] Dropping connectivity from guesagent to 
Trove back-end

Thanks for response, Tim.

As i said, it would be confusing situation when database which has no ACL would 
be deployed by Trove with root enabled - this looks very strange since user 
allowed to check if root enabled. I think in this case Conductor should be 
_that_ place which should contain datastore specific logic, which requires 
back-end connectivity.

It would be nice to have consistent instance states for each datastore types 
and version.

Are there any objections about letting conductor deal with it ?



Best regards,
Denis Makogon


2013/12/20 Tim Simpson 
mailto:tim.simp...@rackspace.com>>
Hi Denis,

The plan from the start with Conductor has been to remove any guest connections 
to the database. So any lingering ones are omissions which should be dealt with.

>> Since not each database have root entity (not even ACL at all) it would be 
>> incorrect to report about root enabling on server-side because 
>> server-side(trove-taskmanager) should stay common as it possible.

I agree that in the case of the root call Conductor should have another RPC 
method that gets called by the guest to inform it that the root entity was set.

I also agree that any code that can stay as common as possible between 
datastores should. However I don't agree that trove-taskmanager (by which I 
assume you mean the daemon) has to only be for common functionality.

Thanks,

Tim


From: Denis Makogon [dmako...@mirantis.com<mailto:dmako...@mirantis.com>]
Sent: Friday, December 20, 2013 7:04 AM
To: OpenStack Development Mailing List
Subject: [openstack-dev] [trove] Dropping connectivity from guesagent to Trove 
back-end


Goodday, OpenStack DВaaS community.



I'd like to start conversation about dropping connectivity from In-VM 
guestagent and Trove back-end.

Since Trove has conductor service which interacts with agents via MQ 
service, we could let it deal with any back-end required operations initialized 
by guestagent.

Now conductor deals with instance status notifications and backup status 
notifications. But guest still have one more operation which requires back-end 
connectivity – database root-enabled reporting [1]. After dealing with it we 
could finally drop connectivity [2].

Since not each database have root entity (not even ACL at all) it would be 
incorrect to report about root enabling on server-side because 
server-side(trove-taskmanager) should stay common as it possible.

My first suggestion was to extend conductor API [3] to let conductor write 
report to Trove back-end. Until Trove would reach state when it would support 
multiple datastore (databases) types current patch would work fine [4], but 
when Trove would deliver, suppose, Database (without ACL) it

Re: [openstack-dev] [trove] Dropping connectivity from guesagent to Trove back-end

[Tim Simpson]

Because the ability to check if root is enabled is in an extension which would 
not be in effect for a datastore with no ACL support, the user would not be 
able to see that the marker for root enabled was set in the Trove 
infrastructure database either way.

By the way- I double checked the code, and I was wrong- the guest agent was 
*not* telling the database to update the root enabled flag. Instead, the API 
extension had been updating the database all along after contacting the guest. 
Sorry for making this thread more confusing.

It seems like if you follow my one (hopefully last) suggestion on this pull 
request, it will solve the issue you're tackling: 
https://review.openstack.org/#/c/59410/5

Thanks,

Tim


From: Denis Makogon [dmako...@mirantis.com]
Sent: Friday, December 20, 2013 8:58 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [trove] Dropping connectivity from guesagent to 
Trove back-end

Thanks for response, Tim.

As i said, it would be confusing situation when database which has no ACL would 
be deployed by Trove with root enabled - this looks very strange since user 
allowed to check if root enabled. I think in this case Conductor should be 
_that_ place which should contain datastore specific logic, which requires 
back-end connectivity.

It would be nice to have consistent instance states for each datastore types 
and version.

Are there any objections about letting conductor deal with it ?



Best regards,
Denis Makogon


2013/12/20 Tim Simpson 
mailto:tim.simp...@rackspace.com>>
Hi Denis,

The plan from the start with Conductor has been to remove any guest connections 
to the database. So any lingering ones are omissions which should be dealt with.

>> Since not each database have root entity (not even ACL at all) it would be 
>> incorrect to report about root enabling on server-side because 
>> server-side(trove-taskmanager) should stay common as it possible.

I agree that in the case of the root call Conductor should have another RPC 
method that gets called by the guest to inform it that the root entity was set.

I also agree that any code that can stay as common as possible between 
datastores should. However I don't agree that trove-taskmanager (by which I 
assume you mean the daemon) has to only be for common functionality.

Thanks,

Tim


From: Denis Makogon [dmako...@mirantis.com<mailto:dmako...@mirantis.com>]
Sent: Friday, December 20, 2013 7:04 AM
To: OpenStack Development Mailing List
Subject: [openstack-dev] [trove] Dropping connectivity from guesagent to Trove 
back-end


Goodday, OpenStack DВaaS community.



I'd like to start conversation about dropping connectivity from In-VM 
guestagent and Trove back-end.

Since Trove has conductor service which interacts with agents via MQ 
service, we could let it deal with any back-end required operations initialized 
by guestagent.

Now conductor deals with instance status notifications and backup status 
notifications. But guest still have one more operation which requires back-end 
connectivity – database root-enabled reporting [1]. After dealing with it we 
could finally drop connectivity [2].

Since not each database have root entity (not even ACL at all) it would be 
incorrect to report about root enabling on server-side because 
server-side(trove-taskmanager) should stay common as it possible.

My first suggestion was to extend conductor API [3] to let conductor write 
report to Trove back-end. Until Trove would reach state when it would support 
multiple datastore (databases) types current patch would work fine [4], but 
when Trove would deliver, suppose, Database (without ACL) it would be confusing 
when after instance provisioning user will find out that some how root was 
enabled, but Database doesn't have any ACL at all.

My point is that Trove Conductor must handle every database (datastore in 
terms of Trove) specific operations which are required back-end connection. And 
Trove server-side (taskmanager) must stay generic and perform preparation 
tasks, which are independent from datastore type.


[1] https://github.com/openstack/trove/blob/master/bin/trove-guestagent#L52

[2] https://bugs.launchpad.net/trove/+bug/1257489

[3] https://review.openstack.org/#/c/59410/5

[4] https://review.openstack.org/#/c/59410/

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [trove] Dropping connectivity from guesagent to Trove back-end

[Tim Simpson]

Hi Denis,

The plan from the start with Conductor has been to remove any guest connections 
to the database. So any lingering ones are omissions which should be dealt with.

>> Since not each database have root entity (not even ACL at all) it would be 
>> incorrect to report about root enabling on server-side because 
>> server-side(trove-taskmanager) should stay common as it possible.

I agree that in the case of the root call Conductor should have another RPC 
method that gets called by the guest to inform it that the root entity was set.

I also agree that any code that can stay as common as possible between 
datastores should. However I don't agree that trove-taskmanager (by which I 
assume you mean the daemon) has to only be for common functionality.

Thanks,

Tim


From: Denis Makogon [dmako...@mirantis.com]
Sent: Friday, December 20, 2013 7:04 AM
To: OpenStack Development Mailing List
Subject: [openstack-dev] [trove] Dropping connectivity from guesagent to Trove 
back-end


Goodday, OpenStack DВaaS community.



I'd like to start conversation about dropping connectivity from In-VM 
guestagent and Trove back-end.

Since Trove has conductor service which interacts with agents via MQ 
service, we could let it deal with any back-end required operations initialized 
by guestagent.

Now conductor deals with instance status notifications and backup status 
notifications. But guest still have one more operation which requires back-end 
connectivity – database root-enabled reporting [1]. After dealing with it we 
could finally drop connectivity [2].

Since not each database have root entity (not even ACL at all) it would be 
incorrect to report about root enabling on server-side because 
server-side(trove-taskmanager) should stay common as it possible.

My first suggestion was to extend conductor API [3] to let conductor write 
report to Trove back-end. Until Trove would reach state when it would support 
multiple datastore (databases) types current patch would work fine [4], but 
when Trove would deliver, suppose, Database (without ACL) it would be confusing 
when after instance provisioning user will find out that some how root was 
enabled, but Database doesn't have any ACL at all.

My point is that Trove Conductor must handle every database (datastore in 
terms of Trove) specific operations which are required back-end connection. And 
Trove server-side (taskmanager) must stay generic and perform preparation 
tasks, which are independent from datastore type.


[1] https://github.com/openstack/trove/blob/master/bin/trove-guestagent#L52

[2] https://bugs.launchpad.net/trove/+bug/1257489

[3] https://review.openstack.org/#/c/59410/5

[4] https://review.openstack.org/#/c/59410/
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [trove] datastore migration issues

[Tim Simpson]

I second Rob and Greg- we need to not allow the instance table to have nulls 
for the datastore version ID. I can't imagine that as Trove grows and evolves, 
that edge case is something we'll always remember to code and test for, so 
let's cauterize things now by no longer allowing it at all.

The fact that the migration scripts can't, to my knowledge, accept parameters 
for what the dummy datastore name and version should be isn't great, but I 
think it would be acceptable enough to make the provided default values 
sensible and ask operators who don't like it to manually update the database.

- Tim




From: Robert Myers [myer0...@gmail.com]
Sent: Thursday, December 19, 2013 9:59 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [trove] datastore migration issues

I think that we need to be good citizens and at least add dummy data. Because 
it is impossible to know who all is using this, the list you have is probably 
complete. But Trove has been available for quite some time and all these users 
will not be listening on this thread. Basically anytime you have a database 
migration that adds a required field you *have* to alter the existing rows. If 
we don't we're basically telling everyone who upgrades that we the 'Database as 
a Service' team don't care about data integrity in our own product :)

Robert


On Thu, Dec 19, 2013 at 9:25 AM, Greg Hill 
mailto:greg.h...@rackspace.com>> wrote:
We did consider doing that, but decided it wasn't really any different from the 
other options as it required the deployer to know to alter that data.  That 
would require the fewest code changes, though.  It was also my understanding 
that mysql variants were a possibility as well (percona and mariadb), which is 
what brought on the objection to just defaulting in code.  Also, we can't 
derive the version being used, so we *could* fill it with a dummy version and 
assume mysql, but I don't feel like that solves the problem or the objections 
to the earlier solutions.  And then we also have bogus data in the database.

Since there's no perfect solution, I'm really just hoping to gather consensus 
among people who are running existing trove installations and have yet to 
upgrade to the newer code about what would be easiest for them.  My 
understanding is that list is basically HP and Rackspace, and maybe Ebay?, but 
the hope was that bringing the issue up on the list might confirm or refute 
that assumption and drive the conversation to a suitable workaround for those 
affected, which hopefully isn't that many organizations at this point.

The options are basically:

1. Put the onus on the deployer to correct existing records in the database.
2. Have the migration script put dummy data in the database which you have to 
correct.
3. Put the onus on the deployer to fill out values in the config value

Greg

On Dec 18, 2013, at 8:46 PM, Robert Myers 
mailto:myer0...@gmail.com>> wrote:


There is the database migration for datastores. We should add a function to  
back fill the existing data with either a dummy data or set it to 'mysql' as 
that was the only possibility before data stores.

On Dec 18, 2013 3:23 PM, "Greg Hill" 
mailto:greg.h...@rackspace.com>> wrote:
I've been working on fixing a bug related to migrating existing installations 
to the new datastore code:

https://bugs.launchpad.net/trove/+bug/1259642

The basic gist is that existing instances won't have any data in the 
datastore_version_id field in the database unless we somehow populate that data 
during migration, and not having that data populated breaks a lot of things 
(including the ability to list instances or delete or resize old instances).  
It's impossible to populate that data in an automatic, generic way, since it's 
highly vendor-dependent on what database and version they currently support, 
and there's not enough data in the older schema to populate the new tables 
automatically.

So far, we've come up with some non-optimal solutions:

1. The first iteration was to assume 'mysql' as the database manager on 
instances without a datastore set.
2. The next iteration was to make the default value be configurable in 
trove.conf, but default to 'mysql' if it wasn't set.
3. It was then proposed that we could just use the 'default_datastore' value 
from the config, which may or may not be set by the operator.

My problem with any of these approaches beyond the first is that requiring 
people to populate config values in order to successfully migrate to the newer 
code is really no different than requiring them to populate the new database 
tables with appropriate data and updating the existing instances with the 
appropriate values.  Either way, it's now highly dependent on people deploying 
the upgrade to know about this change and react accordingly.

Does anyone have a better solution that we aren't considering?  Is this even 
worth the effort given that trove has s

Re: [openstack-dev] [Heat] [Trove] [Savanna] [Oslo] Unified Agents - what is the actual problem?

[Tim Simpson]

>> I agree that enabling communication between guest and cloud service is a 
>> common problem for most agent designs. The only exception is agent based on 
>> hypervisor provided transport. But as far as I understand many people are 
>> interested in network-based agent, so indeed we can start a thread (or 
>> continue discussion in this on) on the problem.

Can't they co-exist?

Let's say the interface to talk to an agent is simply some class loaded from a 
config file, the way it is in Trove. So we have a class which has the methods 
"add_user", "get_filesystem_stats". 

The first, and let's say default, implementation sends a message over Rabbit 
using oslo.rpc or something like it. All the arguments turn into a JSON object 
and are deserialized on the agent side using oslo.rpc or some C++ code capable 
of reading JSON.

If someone wants to add a hypervisor provided transport, they could do so by 
instead changing this API class to one which contacts a service on the 
hypervisor node (using oslo.rpc) with arguments that include the guest agent ID 
and "args", which is just a dictionary of the original arguments. This service 
would then shell out to execute some hypervisor specific command to talk to the 
given guest.

That's what I meant when I said I liked how Trove handled this now- because it 
uses a simple, non-prescriptive interface, it's easy to swap out yet still easy 
to use.

That would mean the job of a unified agent framework would be to offer up 
libraries to ease up the creation of the "API" class by offering Python code to 
send messages in various styles / formats, as well as Python or C++ code to 
read and interpret those messages. 

Of course, we'd still settle on one "default" (probably network based) which 
would become the standard way of sending messages to guests so that package 
maintainers, the Infra team, and newbies to OpenStack wouldn't have to deal 
with dozens of different ways of doing things, but the important thing is that 
other methods of communication would still be possible.

Thanks,

Tim


From: Dmitry Mescheryakov [mailto:dmescherya...@mirantis.com] 
Sent: Thursday, December 19, 2013 7:15 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Heat] [Trove] [Savanna] [Oslo] Unified Agents - 
what is the actual problem?

I agree that enabling communication between guest and cloud service is a common 
problem for most agent designs. The only exception is agent based on hypervisor 
provided transport. But as far as I understand many people are interested in 
network-based agent, so indeed we can start a thread (or continue discussion in 
this on) on the problem.

Dmitry

2013/12/19 Clint Byrum 
So I've seen a lot of really great discussion of the unified agents, and
it has made me think a lot about the problem that we're trying to solve.

I just wanted to reiterate that we should be trying to solve real problems
and not get distracted by doing things "right" or even "better".

I actually think there are three problems to solve.

* Private network guest to cloud service communication.
* Narrow scope highly responsive lean guest agents (Trove, Savanna).
* General purpose in-instance management agent (Heat).

Since the private network guests problem is the only one they all share,
perhaps this is where the three projects should collaborate, and the
other pieces should be left to another discussion.

Thoughts?

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [trove] My thoughts on the Unified Guest Agent

[Tim Simpson]

>> python:
>> consumes 12.5MB of virt memory and 4.3MB of resident memory.

Very few Python only processes will take up such a small amount of virtual 
memory unless the authors are disciplined about not pulling in any dependencies 
at all and writing code in a way that isn't necessarily idiomatic. For an 
example of a project where code is simply written the obvious way, take the 
Trove reference guest, which uses just shy of 200MB of virtual memory and 39MB 
of resident. In defense of the reference guest,  there are some things we can 
do there to make that figure better, I'm certain. I just want to use it as an 
example of how large a Python process can get when the authors proceed doing 
things the way they normally would. 

>> C:
>> 4MB of virt memory and 328k of resident memory

>> C++:
>> 12.5MB of virt memory and 784k of resident memory

Much of the space you're seeing is from the C++ standard library. Building a 
process normally, I get similar results. However it is also possible to 
statically link the standard libraries and knock off roughly half of that, to 
6.42MB virtual and 400kb resident.

Additionally, the C++ standard library can be omitted if necessary. At this 
point, you might argue that you'd just be writing C code, but even then you'd 
have the advantages of template metaprogramming and other features not present 
in plain C. Even without those features, there's no shame in writing C style 
code assuming you *have* to- C++ was designed to be compatible with C to take 
advantages of its strengths. The only loss would be some of the C99 stuff like 
named initializers.

Additionally, in a vast number of contexts the virtual memory "used" for the 
standard library is not going to matter as other processes will be including 
that code anyway.

Going back to the Trove C++ Agent, it takes 4MB of resident and 28MB of virtual 
memory. This is with some fairly non-trivial dependencies, such as Curl, libz, 
the MySQL and Rabbit client libraries. No special effort was expended making 
sure we kept the process small as in C++ things naturally stay tiny.

>> C++ is full of fail in a variety of ways and offers no useful advantage for 
>> something as small as an agent ;-)

If you haven't recently, I recommend you read up on modern C++. The language, 
and how it's written and explained, has changed a lot over the past ten years.

Thanks,

Tim


-Original Message-
From: Steven Dake [mailto:sd...@redhat.com] 
Sent: Wednesday, December 18, 2013 4:15 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [trove] My thoughts on the Unified Guest Agent

On 12/18/2013 12:27 PM, Tim Simpson wrote:
>>> Please provide proof of that assumption or at least a general hypothesis 
>>> that we can test.
> I can't prove that the new agent will be larger as it doesn't exist yet.
>
>>> Since nothing was agreed upon anyway, I don't know how you came to that 
>>> conclusion.  I would suggest that any agent framework be held to an 
>>> extremely high standard for footprint for this very reason.
> Sorry, I formed a conclusion based on what I'd read so far. There has been 
> talk to add Salt to this Unified Agent along with several other things. So I 
> think its a valid concern to state that making this thing small is not as 
> high on the list of priorities as adding extra functionality.
>
> The C++ agent is just over 3 megabytes of real memory and takes up less than 
> 30 megabytes  of virtual memory. I don't think an agent has to be *that* 
> small. However it won't get near that small unless making it tiny is made a 
> priority, and I'm skeptical that's possible while also deciding an agent will 
> be capable of interacting with all major OpenStack projects as well as Salt.
>
>>> Nobody has suggested writing an agent that does everything.
> Steven Dake just said:
>
> "A unified agent addresses the downstream viewpoint well, which is 'There is 
> only one agent to package and maintain, and it supports all the integrated 
> OpenStack Program projects'."
>
> So it sounds like some people are saying there will only be one. Or that it 
> is at least an idea.
>
>>> If Trove's communication method is in fact superior to all others, then 
>>> perhaps we should discuss using that in the unified agent framework.
> My point is every project should communicate to an agent in its own 
> interface, which can be swapped out for whatever implementations people need.
>
>>>   In fact I've specifically been arguing to keep it focused on facilitating 
>>> guest<->service communication and limiting its in-guest capabilities to 
>>> narrowly focu

Re: [openstack-dev] [trove] My thoughts on the Unified Guest Agent

[Tim Simpson]

Thanks for the summary Dmitry. I'm ok with these ideas, and while I still 
disagree with having a single, forced standard for RPC communication, I should 
probably let things pan out a bit before being too concerned.

- Tim



From: Dmitry Mescheryakov [dmescherya...@mirantis.com]
Sent: Wednesday, December 18, 2013 11:51 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [trove] My thoughts on the Unified Guest Agent

Tim,

The unified agent we proposing is based on the following ideas:
  * the core agent has _no_ functionality at all. It is a pure RPC mechanism 
with the ability to add whichever API needed on top of it.
  * the API is organized into modules which could be reused across different 
projects.
  * there will be no single package: each project (Trove/Savanna/Others) 
assembles its own agent based on API project needs.

I hope that covers your concerns.

Dmitry


2013/12/18 Tim Simpson 
mailto:tim.simp...@rackspace.com>>
I've been following the Unified Agent mailing list thread for awhile now and, 
as someone who has written a fair amount of code for both of the two existing 
Trove agents, thought I should give my opinion about it. I like the idea of a 
unified agent, but believe that forcing Trove to adopt this agent for use as 
its by default will stifle innovation and harm the project.

There are reasons Trove has more than one agent currently. While everyone knows 
about the "Reference Agent" written in Python, Rackspace uses a different agent 
written in C++ because it takes up less memory. The concerns which led to the 
C++ agent would not be addressed by a unified agent, which if anything would be 
larger than the Reference Agent is currently.

I also believe a unified agent represents the wrong approach philosophically. 
An agent by design needs to be lightweight, capable of doing exactly what it 
needs to and no more. This is especially true for a project like Trove whose 
goal is to not to provide overly general PAAS capabilities but simply 
installation and maintenance of different datastores. Currently, the Trove 
daemons handle most logic and leave the agents themselves to do relatively 
little. This takes some effort as many of the first iterations of Trove 
features have too much logic put into the guest agents. However through 
perseverance the subsequent designs are usually cleaner and simpler to follow. 
A community approved, "do everything" agent would endorse the wrong balance and 
lead to developers piling up logic on the guest side. Over time, features would 
become dependent on the Unified Agent, making it impossible to run or even 
contemplate light-weight agents.

Trove's interface to agents today is fairly loose and could stand to be made 
stricter. However, it is flexible and works well enough. Essentially, the duck 
typed interface of the trove.guestagent.api.API class is used to send messages, 
and Trove conductor is used to receive them at which point it updates the 
database. Because both of these components can be swapped out if necessary, the 
code could support the Unified Agent when it appears as well as future agents.

It would be a mistake however to alter Trove's standard method of communication 
to please the new Unified Agent. In general, we should try to keep Trove 
speaking to guest agents in Trove's terms alone to prevent bloat.

Thanks,

Tim

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [trove] My thoughts on the Unified Guest Agent

[Tim Simpson]

>> Please provide proof of that assumption or at least a general hypothesis 
>> that we can test.

I can't prove that the new agent will be larger as it doesn't exist yet. 

>> Since nothing was agreed upon anyway, I don't know how you came to that 
>> conclusion.  I would suggest that any agent framework be held to an 
>> extremely high standard for footprint for this very reason.

Sorry, I formed a conclusion based on what I'd read so far. There has been talk 
to add Salt to this Unified Agent along with several other things. So I think 
its a valid concern to state that making this thing small is not as high on the 
list of priorities as adding extra functionality. 

The C++ agent is just over 3 megabytes of real memory and takes up less than 30 
megabytes  of virtual memory. I don't think an agent has to be *that* small. 
However it won't get near that small unless making it tiny is made a priority, 
and I'm skeptical that's possible while also deciding an agent will be capable 
of interacting with all major OpenStack projects as well as Salt.

>> Nobody has suggested writing an agent that does everything.

Steven Dake just said:

"A unified agent addresses the downstream viewpoint well, which is 'There is 
only one agent to package and maintain, and it supports all the integrated 
OpenStack Program projects'."

So it sounds like some people are saying there will only be one. Or that it is 
at least an idea.

>> If Trove's communication method is in fact superior to all others, then 
>> perhaps we should discuss using that in the unified agent framework.

My point is every project should communicate to an agent in its own interface, 
which can be swapped out for whatever implementations people need.

>>  In fact I've specifically been arguing to keep it focused on facilitating 
>> guest<->service communication and limiting its in-guest capabilities to 
>> narrowly focused tasks.

I like this idea better than creating one agent to rule them all, but I would 
like to avoid forcing a single method of communicating between agents.

>> Also I'd certainly be interested in hearing about whether or not you think 
>> the C++ agent could made generic enough for any project to use.

I certainly believe much of the code could be reused for other projects. Right 
now it communicates over RabbitMQ, Oslo RPC style, so I'm not sure how much it 
will fall in line with what the Unified Agent group wants. However, I would 
love to talk more about this. So far my experience has been that no one wants 
to pursue using / developing an agent that was written in C++.

Thanks,

Tim

From: Clint Byrum [cl...@fewbar.com]
Sent: Wednesday, December 18, 2013 11:36 AM
To: openstack-dev
Subject: Re: [openstack-dev] [trove] My thoughts on the Unified Guest Agent

Excerpts from Tim Simpson's message of 2013-12-18 07:34:14 -0800:
> I've been following the Unified Agent mailing list thread for awhile
> now and, as someone who has written a fair amount of code for both of
> the two existing Trove agents, thought I should give my opinion about
> it. I like the idea of a unified agent, but believe that forcing Trove
> to adopt this agent for use as its by default will stifle innovation
> and harm the project.
>

"Them's fightin words". ;)

That is a very strong position to take. So I am going to hold your
statements of facts and assumptions to a very high standard below.

> There are reasons Trove has more than one agent currently. While
> everyone knows about the "Reference Agent" written in Python, Rackspace
> uses a different agent written in C++ because it takes up less memory. The
> concerns which led to the C++ agent would not be addressed by a unified
> agent, which if anything would be larger than the Reference Agent is
> currently.
>

"Would be larger..." - Please provide proof of that assumption or at least
a general hypothesis that we can test. Since nothing was agreed upon
anyway, I don't know how you came to that conclusion. I would suggest
that any agent framework be held to an extremely high standard for
footprint for this very reason.

> I also believe a unified agent represents the wrong approach
> philosophically. An agent by design needs to be lightweight, capable
> of doing exactly what it needs to and no more. This is especially true
> for a project like Trove whose goal is to not to provide overly general
> PAAS capabilities but simply installation and maintenance of different
> datastores. Currently, the Trove daemons handle most logic and leave
> the agents themselves to do relatively little. This takes some effort
> as many of the first iterations of Trove features have too much logic
> put into the guest agents. However through perseverance the subsequent
> designs are usually cleaner and simpler to follow. A community approved,
> "do everything" agent would endorse the wrong balance and lead to
> developers piling up logic on the guest side. Over time, features would
> becom

[openstack-dev] [trove] My thoughts on the Unified Guest Agent

[Tim Simpson]

I've been following the Unified Agent mailing list thread for awhile now and, 
as someone who has written a fair amount of code for both of the two existing 
Trove agents, thought I should give my opinion about it. I like the idea of a 
unified agent, but believe that forcing Trove to adopt this agent for use as 
its by default will stifle innovation and harm the project.

There are reasons Trove has more than one agent currently. While everyone knows 
about the "Reference Agent" written in Python, Rackspace uses a different agent 
written in C++ because it takes up less memory. The concerns which led to the 
C++ agent would not be addressed by a unified agent, which if anything would be 
larger than the Reference Agent is currently.

I also believe a unified agent represents the wrong approach philosophically. 
An agent by design needs to be lightweight, capable of doing exactly what it 
needs to and no more. This is especially true for a project like Trove whose 
goal is to not to provide overly general PAAS capabilities but simply 
installation and maintenance of different datastores. Currently, the Trove 
daemons handle most logic and leave the agents themselves to do relatively 
little. This takes some effort as many of the first iterations of Trove 
features have too much logic put into the guest agents. However through 
perseverance the subsequent designs are usually cleaner and simpler to follow. 
A community approved, "do everything" agent would endorse the wrong balance and 
lead to developers piling up logic on the guest side. Over time, features would 
become dependent on the Unified Agent, making it impossible to run or even 
contemplate light-weight agents.

Trove's interface to agents today is fairly loose and could stand to be made 
stricter. However, it is flexible and works well enough. Essentially, the duck 
typed interface of the trove.guestagent.api.API class is used to send messages, 
and Trove conductor is used to receive them at which point it updates the 
database. Because both of these components can be swapped out if necessary, the 
code could support the Unified Agent when it appears as well as future agents.

It would be a mistake however to alter Trove's standard method of communication 
to please the new Unified Agent. In general, we should try to keep Trove 
speaking to guest agents in Trove's terms alone to prevent bloat.

Thanks,

Tim
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Trove] How users should specify a datastore type when creating an instance

[Tim Simpson]

ther disambiguation _is_ needed.

6. trove-cli instance create --datastore_type redis --datastore_version 2.6.16
We have both datastore_type and datastore_version, and that uniquely
identifies redis 2.6.16 . No further disambiguation is needed.

7. trove-cli instance create --datastore_type cassandra --version 2.0.0,
or trove-cli instance create --datastore_id g
Here, we are attempting to deploy a datastore which is _NOT_ active and
this call should fail with an appropriate error message.

Cheers,
-Nikhil


Andrey Shestakov writes:

> 2. it can be confusing coz not clear to what type version belongs
> (possible add "type" field in version).
> also if you have default type, then specified version recognizes as
> version of default type (no lookup in version.datastore_type_id)
> but i think we can do lookup in version.datastore_type_id before pick
> default.
>
> 4. if default version is need, then it should be specified in db, coz
> switching via versions can be frequent and restart service to reload
> config all times is not good.
>
> On 10/21/2013 05:12 PM, Tim Simpson wrote:
>> Thanks for the feedback Andrey.
>>
>> >> 2. Got this case in irc, and decided to pass type and version
>> together to avoid confusing.
>> I don't understand how allowing the user to only pass the version
>> would confuse anyone. Could you elaborate?
>>
>> >> 3. Names of types and maybe versions can be good, but in irc conversation 
>> >> rejected this case, i cant
>> remember exactly reason.
>> Hmm. Does anyone remember the reason for this?
>>
>> >> 4. Actually, "active" field in version marks it as default in type.
>> >>Specify default version in config can be usefull if you have more then
>> one active versions in default type.
>> If 'active' is allowed to be set for multiple rows of the
>> 'datastore_versions' table then it isn't a good substitute for the
>> functionality I'm seeking, which is to allow operators to specify a
>> *single* default version for each datastore_type in the database. I
>> still think we should still add a 'default_version_id' field to the
>> 'datastore_types' table.
>>
>> Thanks,
>>
>> Tim
>>
>> 
>> *From:* Andrey Shestakov 
>> [ashesta...@mirantis.com<mailto:ashesta...@mirantis.com>]
>> *Sent:* Monday, October 21, 2013 7:15 AM
>> *To:* OpenStack Development Mailing List
>> *Subject:* Re: [openstack-dev] [Trove] How users should specify a
>> datastore type when creating an instance
>>
>> 1. Good point
>> 2. Got this case in irc, and decided to pass type and version together
>> to avoid confusing.
>> 3. Names of types and maybe versions can be good, but in irc
>> conversation rejected this case, i cant remember exactly reason.
>> 4. Actually, "active" field in version marks it as default in type.
>> Specify default version in config can be usefull if you have more then
>> one active versions in default type.
>> But how match active version in type depends on operator`s
>> configuration. And what if "default version in config" will marked as
>> inactive?
>>
>> On 10/18/2013 10:30 PM, Tim Simpson wrote:
>>> Hello fellow Trovians,
>>>
>>> There has been some good work recently to figure out a way to specify
>>> a specific datastore  when using Trove. This is essential to
>>> supporting multiple datastores from the same install of Trove.
>>>
>>> I have an issue with some elements of the proposed solution though,
>>> so I decided I'd start a thread here so we could talk about it.
>>>
>>> As a quick refresher, here is the blue print for this work (there are
>>> some gists ammended to the end but I figured the mailing list would
>>> be an easier venue for discussion):
>>> https://wiki.openstack.org/wiki/Trove/trove-versions-types
>>>
>>> One issue I have is with the way the instance create call will change
>>> to support different data stores. For example, here is the post call:
>>>
>>> """
>>> {
>>>   "instance" : {
>>>   "flavorRef" : "2",
>>>   "name" : "as",
>>>   "datastore_type" : "e60153d4-8ac4-414a-ad58-fe2e0035704a",
>>>   "datastore_version" : "94ed1f9f-6c1a-4d6e-87e9-04ecff37b64b",
>>>   "volume" : { &quo

Re: [openstack-dev] [Trove] How users should specify a datastore type when creating an instance

[Tim Simpson]

So if we decide to support any number of config options for each various 
datastore version, eventually we'll have large config files that will be hard 
to manage.

What about storing the extra config info for each datastore version in its own 
independent config file? So rather than having one increasingly bloated config 
file used by everything, you could optionally specify a file in the 
datastore_versions table of the database that would be looked up similar to how 
we load template files on demand.

- Tim

From: Ilya Sviridov [isviri...@mirantis.com]
Sent: Thursday, October 24, 2013 7:40 AM
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] [Trove] How users should specify a datastore type 
when creating an instance

So, we have 2 places for configuration management - database and config file

Config file for tunning all datasource type behavior during installation and 
database for all changeable configurations during usage and administration of 
Trove installation.

Database usecases:
- update/custom image
- update/custom packages
- activating/deactivating datastore_type

Config file usecases:
- security group policy
- provisioning mechanism
- guest configuration parameters per database engine
- provisioning  parameters, templates
- manager class
...

In case if i need to register one more MySQL installation with following 
customization:
- custom heat template
- custom packages and additional monitoring tool package
- open specific port for working with my monitoring tool on instance

According to current concept should i add one more section in addition to 
existing mysql like below?

[monitored_mysql]
mount_point=/var/lib/mysql

#8080 is port of my monitoring tool
trove_security_group_rule_ports = 3306, 8080
heat_template=/etc/trove/heat_templates/monitored_mysql.yaml
...

and put additional packages to database configuration?





With best regards,
Ilya Sviridov




On Wed, Oct 23, 2013 at 9:37 PM, Michael Basnight 
mailto:mbasni...@gmail.com>> wrote:

On Oct 23, 2013, at 10:54 AM, Ilya Sviridov wrote:

> Besides the strategy of selecting the default behavior.
>
> Let me share with you my ideas of configuration management in Trove and how 
> the datastore concept can help with that.
>
> Initially there was only one database and all configuration was in one config 
> file.
> With adding of new databases, heat provisioning mechanism, we are introducing 
> more options.
>
> Not only assigning specific image_id, but custom packages, heat templates, 
> probably specific strategies of working with security groups.
> Such needs already exist because we have a lot of optional things in config, 
> and any new feature is implemented with back sight to already existing legacy 
> installations of Trove.
>
> What is  actually datastore_type + datastore_version?
>
> The model which glues all the bricks together, so let us use it for all 
> variable part of *service type* configuration.
>
> from current config file
>
> # Trove DNS
> trove_dns_support = False
>
> # Trove Security Groups for Instances
> trove_security_groups_support = True
> trove_security_groups_rules_support = False
> trove_security_group_rule_protocol = tcp
> trove_security_group_rule_port = 3306
> trove_security_group_rule_cidr = 0.0.0.0/0
>
> #guest_config = $pybasedir/etc/trove/trove-guestagent.conf.sample
> #cloudinit_location = /etc/trove/cloudinit
>
> block_device_mapping = vdb
> device_path = /dev/vdb
> mount_point = /var/lib/mysql
>
> All that configurations can be moved to data_strore (some defined in heat 
> templates) and be manageable by operator in case if any default behavior 
> should be changed.
>
> The trove-config becomes core functionality specific only.

Its fine for it to be in the config or the heat templates… im not sure it 
matters. what i would like to see is that specific thing to each service be in 
their own config group in the configuration.

[mysql]
mount_point=/var/lib/mysql
…
[redis]
volume_support=False
…..

and so on.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Trove] How users should specify a datastore type when creating an instance

[Tim Simpson]

> It's not intuitive to the User, if they are specifying a version alone.  You 
> don't boot a 'version' of something, with specifying what that some thing is. 
>  I would rather they only specified the datastore_type alone, and not have 
> them specify a version at all.

I agree for most users just selecting the datastore_type would be most intutive.

However, when they specify a version it's going to a be GUID which they could 
only possibly know if they have recently enumerated all versions and thus 
*know* the version is for the given type they want. In that case I don't think 
most users would appreciate having to also pass the type- it would just be 
redundant. So in that case why not make it optional?


From: Vipul Sabhaya [vip...@gmail.com]
Sent: Monday, October 21, 2013 5:09 PM
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] [Trove] How users should specify a datastore type 
when creating an instance




On Mon, Oct 21, 2013 at 2:04 PM, Michael Basnight 
mailto:mbasni...@gmail.com>> wrote:

On Oct 21, 2013, at 1:40 PM, Tim Simpson wrote:

>>> 2. I also think a datastore_version alone should be sufficient since the 
>>> associated datastore type will be implied:
>
>> When i brought this up it was generally discussed as being confusing. Id 
>> like to use type and rely on having a default (or active) version behind the 
>> scenes.
>
> Can't we do both? If a user wants a specific version, most likely they had to 
> enumerate all datastore_versions, spot it in a list, and grab the guid. Why 
> force them to also specify the datastore_type when we can easily determine 
> what that is?

Fair enough.


It's not intuitive to the User, if they are specifying a version alone.  You 
don't boot a 'version' of something, with specifying what that some thing is.  
I would rather they only specified the datastore_type alone, and not have them 
specify a version at all.

>
>>> 4. Additionally, in the current pull request to implement this it is 
>>> possible to avoid passing a version, but only if no more than one version 
>>> of the datastore_type exists in the database.
>>>
>>> I think instead the datastore_type row in the database should also have a 
>>> "default_version_id" property, that an operator could update to the most 
>>> recent version or whatever other criteria they wish to use, meaning the 
>>> call could become this simple:
>
>> Since we have determined from this email thread that we have an active 
>> status, and that > 1 version can be active, we have to think about the 
>> precedence of active vs default. My question would be, if we have a 
>> default_version_id and a active version, what do we choose on behalf of the 
>> user? If there is > 1 active version and a user does not specify the 
>> version, the api will error out, unless a default is defined. We also need a 
>> default_type in the config so the existing APIs can maintain compatibility. 
>> We can re-discuss this for v2 of the API.
>
> Imagine that an operator sets up Trove and only has one active version. They 
> then somehow fumble setting up the default_version, but think they succeeded 
> as the API works for users the way they expect anyway. Then they go to add 
> another active version and suddenly their users get error messages.
>
> If we only use the "default_version" field of the datastore_type to define a 
> default would honor the principle of least surprise.

Are you saying you must have a default version defined to have > 1 active 
versions?


I think it makes sense to have a 'Active' flag on every version -- and a 
default flag for the version that should be used as a default in the event the 
user doesn't specify.  It also makes sense to require the deployer to set this 
accurately, and if one doesn't exist instance provisioning errors out.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Trove] How users should specify a datastore type when creating an instance

[Tim Simpson]

>  i think we have to use default version only if type contains > 1
True, but a default should be settable even if there's only one type.

> And default version should be active, otherwise -
> error.
Agreed. 


From: Andrey Shestakov [ashesta...@mirantis.com]
Sent: Monday, October 21, 2013 4:40 PM
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] [Trove] How users should specify a datastore type 
when creating an instance

On Mon, Oct 21, 2013 at 11:40 PM, Tim Simpson  wrote:
>
> >> 4. Additionally, in the current pull request to implement this it is 
> >> possible to avoid passing a version, but only if no more than one version 
> >> of the datastore_type exists in the database.
> >>
> >> I think instead the datastore_type row in the database should also have a 
> >> "default_version_id" property, that an operator could update to the most 
> >> recent version or whatever other criteria they wish to use, meaning the 
> >> call could become this simple:
>
> > Since we have determined from this email thread that we have an active 
> > status, and that > 1 version can be active, we have to think about the 
> > precedence of active vs default. My question would be, if we have a 
> > default_version_id and a active version, what do we choose on behalf of the 
> > user? If there is > 1 active version and a user does not specify the 
> > version, the api will error out, unless a default is defined. We also need 
> > a default_type in the config so the existing APIs can maintain 
> > compatibility. We can re-discuss this for v2 of the API.
>
> Imagine that an operator sets up Trove and only has one active version. They 
> then somehow fumble setting up the default_version, but think they succeeded 
> as the API works for users the way they expect anyway. Then they go to add 
> another active version and suddenly their users get error messages.
>
> If we only use the "default_version" field of the datastore_type to define a 
> default would honor the principle of least surprise.

if default version is inactive? there will more cases for error

also, i think we have to use default version only if type contains > 1
active version. And default version should be active, otherwise -
error.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Trove] How users should specify a datastore type when creating an instance

[Tim Simpson]

> This is also true that we dont want to define the _need_ to have custom 
> images for the datastores. You can, quite easily, deploy mysql or redis on a 
> vanilla image.

Additionally there could be server code at some point soon that will need to 
know what datastore type is associated with an instance to determine what db 
engine is in use. So for example, if a call such as "users" isn't supported by 
a certain datastore used by an instance, the server side code will be able to 
determine that and something such as a bad request or not found status code.


From: Michael Basnight [mbasni...@gmail.com]
Sent: Monday, October 21, 2013 4:05 PM
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] [Trove] How users should specify a datastore   
type when creating an instance

On Oct 21, 2013, at 1:57 PM, Nikhil Manchanda wrote:

>
> The image approach works fine if Trove only supports deploying a single
> datastore type (mysql in your case). As soon as we support
> deploying more than 1 datastore type, Trove needs to have some knowledge
> of which guestagent manager classes to load. Hence the need
> for having a datastore type API.
>
> The argument for needing to keep track of the version is
> similar. Potentially a version increment -- especially of the major
> version -- may require for a different guestagent manager. And Trove
> needs to have this information.

This is also true that we dont want to define the _need_ to have custom images 
for the datastores. You can, quite easily, deploy mysql or redis on a vanilla 
image.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Trove] How users should specify a datastore type when creating an instance

[Tim Simpson]

> Are you saying you must have a default version defined to have > 1 active 
> versions?
No, my point was using a default version field in the db rather than also 
picking from active versions may be confusing.


From: Michael Basnight [mbasni...@gmail.com]
Sent: Monday, October 21, 2013 4:04 PM
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] [Trove] How users should specify a datastore   
type when creating an instance

On Oct 21, 2013, at 1:40 PM, Tim Simpson wrote:

>>> 2. I also think a datastore_version alone should be sufficient since the 
>>> associated datastore type will be implied:
>
>> When i brought this up it was generally discussed as being confusing. Id 
>> like to use type and rely on having a default (or active) version behind the 
>> scenes.
>
> Can't we do both? If a user wants a specific version, most likely they had to 
> enumerate all datastore_versions, spot it in a list, and grab the guid. Why 
> force them to also specify the datastore_type when we can easily determine 
> what that is?

Fair enough.

>
>>> 4. Additionally, in the current pull request to implement this it is 
>>> possible to avoid passing a version, but only if no more than one version 
>>> of the datastore_type exists in the database.
>>>
>>> I think instead the datastore_type row in the database should also have a 
>>> "default_version_id" property, that an operator could update to the most 
>>> recent version or whatever other criteria they wish to use, meaning the 
>>> call could become this simple:
>
>> Since we have determined from this email thread that we have an active 
>> status, and that > 1 version can be active, we have to think about the 
>> precedence of active vs default. My question would be, if we have a 
>> default_version_id and a active version, what do we choose on behalf of the 
>> user? If there is > 1 active version and a user does not specify the 
>> version, the api will error out, unless a default is defined. We also need a 
>> default_type in the config so the existing APIs can maintain compatibility. 
>> We can re-discuss this for v2 of the API.
>
> Imagine that an operator sets up Trove and only has one active version. They 
> then somehow fumble setting up the default_version, but think they succeeded 
> as the API works for users the way they expect anyway. Then they go to add 
> another active version and suddenly their users get error messages.
>
> If we only use the "default_version" field of the datastore_type to define a 
> default would honor the principle of least surprise.

Are you saying you must have a default version defined to have > 1 active 
versions?

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Trove] How users should specify a datastore type when creating an instance

[Tim Simpson]

>> 2. I also think a datastore_version alone should be sufficient since the 
>> associated datastore type will be implied:

>When i brought this up it was generally discussed as being confusing. Id like 
>to use type and rely on having a default (or active) version behind the scenes.

Can't we do both? If a user wants a specific version, most likely they had to 
enumerate all datastore_versions, spot it in a list, and grab the guid. Why 
force them to also specify the datastore_type when we can easily determine what 
that is?

>> 4. Additionally, in the current pull request to implement this it is 
>> possible to avoid passing a version, but only if no more than one version of 
>> the datastore_type exists in the database.
>>
>> I think instead the datastore_type row in the database should also have a 
>> "default_version_id" property, that an operator could update to the most 
>> recent version or whatever other criteria they wish to use, meaning the call 
>> could become this simple:

> Since we have determined from this email thread that we have an active 
> status, and that > 1 version can be active, we have to think about the 
> precedence of active vs default. My question would be, if we have a 
> default_version_id and a active version, what do we choose on behalf of the 
> user? If there is > 1 active version and a user does not specify the version, 
> the api will error out, unless a default is defined. We also need a 
> default_type in the config so the existing APIs can maintain compatibility. 
> We can re-discuss this for v2 of the API.

Imagine that an operator sets up Trove and only has one active version. They 
then somehow fumble setting up the default_version, but think they succeeded as 
the API works for users the way they expect anyway. Then they go to add another 
active version and suddenly their users get error messages.

If we only use the "default_version" field of the datastore_type to define a 
default would honor the principle of least surprise.



From: Michael Basnight [mbasni...@gmail.com]
Sent: Monday, October 21, 2013 3:12 PM
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] [Trove] How users should specify a datastore   
type when creating an instance

On Oct 18, 2013, at 12:30 PM, Tim Simpson wrote:

> 1. I think since we have two fields in the instance object we should make a 
> new object for datastore and avoid the name prefixing, like this:

I agree with this.

> 2. I also think a datastore_version alone should be sufficient since the 
> associated datastore type will be implied:

When i brought this up it was generally discussed as being confusing. Id like 
to use type and rely on having a default (or active) version behind the scenes.

> 3. Additionally, while a datastore_type should have an ID in the Trove 
> infastructure database, it should also be possible to pass just the name of 
> the datastore type to the instance call, such as "mysql" or "mongo". Maybe we 
> could allow this in addition to the ID? I think this form should actually use 
> the argument "type", and the id should then be passed as "type_id" instead.

Id prefer this honestly.

> 4. Additionally, in the current pull request to implement this it is possible 
> to avoid passing a version, but only if no more than one version of the 
> datastore_type exists in the database.
>
> I think instead the datastore_type row in the database should also have a 
> "default_version_id" property, that an operator could update to the most 
> recent version or whatever other criteria they wish to use, meaning the call 
> could become this simple:

Since we have determined from this email thread that we have an active status, 
and that > 1 version can be active, we have to think about the precedence of 
active vs default. My question would be, if we have a default_version_id and a 
active version, what do we choose on behalf of the user? If there is > 1 active 
version and a user does not specify the version, the api will error out, unless 
a default is defined. We also need a default_type in the config so the existing 
APIs can maintain compatibility. We can re-discuss this for v2 of the API.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Trove] Testing of new service types support

[Tim Simpson]

>> For the api stuff, sure thats fine. i just think the overall coverage of the 
>> review will be quite low if we are only testing the API via fake code.

We're in agreement here, I think. I will say though that if the people working 
on Mongo want to test it early, and go beyond simply using the client to 
manually confirm stuff, it should be possible to run the existing tests by 
building a different image and running a subset, such as 
"--group=dbaas.guest.shutdown". IIRC those tests don't do much other than make 
an instance, see it turn to ACTIVE, and delete it. It would be a worthwhile 
spot test to see if it adheres to the bare-minimum Trove API.


From: Michael Basnight [mbasni...@gmail.com]
Sent: Monday, October 21, 2013 12:19 PM
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] [Trove] Testing of new service types support

On Oct 21, 2013, at 10:02 AM, Tim Simpson wrote:

> Can't we say that about nearly any feature though? In theory we could put a 
> hold on any tests for feature work saying it
> will need to be redone when Tempest integrated is finished.
>
> Keep in mind what I'm suggesting here is a fairly trivial change to get some 
> validation via the existing fake mode / integration tests at a fairly small 
> cost.

Of course we can do the old tests. And for this it might be the best thing. The 
problem i see is that we cant do real integration tests w/o this work, and i 
dont want to integrate a bunch of different service_types w/o tests that 
actually spin them up and run the guest, which is where 80% of the "new" code 
lives for a new service_type. Otherwise we are running fake-guest stuff that is 
not a good representation.

For the api stuff, sure thats fine. i just think the overall coverage of the 
review will be quite low if we are only testing the API via fake code.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Trove] Testing of new service types support

[Tim Simpson]

Can't we say that about nearly any feature though? In theory we could put a 
hold on any tests for feature work saying it 
will need to be redone when Tempest integrated is finished.

Keep in mind what I'm suggesting here is a fairly trivial change to get some 
validation via the existing fake mode / integration tests at a fairly small 
cost.


From: Michael Basnight [mbasni...@gmail.com]
Sent: Monday, October 21, 2013 11:45 AM
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] [Trove] Testing of new service types support

Top posting…

Id like to see these in the tempest tests. Im just getting started integrating 
trove into tempest for testing, and there are some prerequisites that im 
working thru with the infra team. Progress is being made though. Id rather not 
see them go into 2 different test suites if we can just get them into the 
tempest tests. Lets hope the stars line up so that you can start testing in 
tempest. :)

On Oct 21, 2013, at 9:25 AM, Illia Khudoshyn wrote:

> Hi Tim,
>
> Thanks for a quick reply. I'll go with updating run_tests.py for now. Hope, 
> Andrey Shestakov's changes arrive soon.
>
> Best wishes.
>
>
>
> On Mon, Oct 21, 2013 at 7:01 PM, Tim Simpson  
> wrote:
> Hi Illia,
>
> You're correct; until the work on establishing datastore types and versions 
> as a first class Trove concept is finished, which will hopefully be soon (see 
> Andrey Shestakov's pull request), testing non-MySQL datastore types will be 
> problematic.
>
> A short term, fake-mode only solution could be accomplished fairly quickly as 
> follows: run the fake mode tests a third time in Tox with a new configuration 
> which allows for MongoDB.
>
> If you look at tox.ini, you'll see that the integration tests run in fake 
> mode twice already:
>
> >> {envpython} run_tests.py
> >> {envpython} run_tests.py --test-config=etc/tests/xml.localhost.test.conf
>
> The second invocation causes the trove-client to be used in XML mode, 
> effectively testing the XML client.
>
> (Tangent: currently running the tests twice takes some time, even in fake 
> mode- however it will cost far less time once the following pull request is 
> merged: https://review.openstack.org/#/c/52490/)
>
> If you look at run_tests.py, you'll see that on line 104 it accepts a trove 
> config file. If the run_tests.py script is updated to allow this value to be 
> specified optionally via the command line, you could create a variation on 
> "etc/trove/trove.conf.test" which specifies MongoDB. You'd then invoke 
> run_tests.py with a "--group=" argument to run some subset of the tests 
> support by the current Mongo DB code in fake mode.
>
> Of course, this will do nothing to test the guest agent changes or confirm 
> that the end to end system actually works, but it could help test a lot of 
> incidental API and infrastructure database code.
>
> As for real mode tests, I think we should wait until the datastore type / 
> version code is finished, at which point I know we'll all be eager to add 
> additional tests for these new datastores. Of course in the short term it 
> should be possible for you to  change the code locally to build a Mongo DB 
> image as well as a Trove config file to support this and then just run some 
> subset of tests that works with Mongo.
>
> Thanks,
>
> Tim
>
>
> From: Illia Khudoshyn [ikhudos...@mirantis.com]
> Sent: Monday, October 21, 2013 9:42 AM
> To: OpenStack Development Mailing List
> Subject: [openstack-dev] [Trove] Testing of new service types support
>
> Hi all,
>
> I've done with implementing the very first bits of MongoDB support in Trove 
> along with unit tests and faced an issue with proper testing of it.
>
> It is well known that right now only one service type per installation is 
> supported by Trove (it is set in config). All testing infrastructure, 
> including Trove-integration codebase and jenkins jobs, seem to rely on that 
> service type as well. So it seems to be impossible to run all existing tests 
> AND some additional tests for MongoDB service type in one pass, at least 
> until Trove client will allow to pass service type (I know that there is 
> ongoing work in this area).
>
> Please note, that all of the above is about functional and intergation 
> testing -- there is no issues with unit tests.
>
> So the question is, should I first submit the code to Trove and then proceed 
> with updating Trove-integration or just put aside all that MongoDB stuff 
> until client (and -integration) will be ready?
>
> PS AFAIK, there is some work on adding Cassandra and Riak (or Redis?) support 
> to Trove. Th

Re: [openstack-dev] [Trove] Testing of new service types support

[Tim Simpson]

Hi Illia,

You're correct; until the work on establishing datastore types and versions as 
a first class Trove concept is finished, which will hopefully be soon (see 
Andrey Shestakov's pull request), testing non-MySQL datastore types will be 
problematic.

A short term, fake-mode only solution could be accomplished fairly quickly as 
follows: run the fake mode tests a third time in Tox with a new configuration 
which allows for MongoDB.

If you look at tox.ini, you'll see that the integration tests run in fake mode 
twice already:

>> {envpython} run_tests.py
>> {envpython} run_tests.py --test-config=etc/tests/xml.localhost.test.conf

The second invocation causes the trove-client to be used in XML mode, 
effectively testing the XML client.

(Tangent: currently running the tests twice takes some time, even in fake mode- 
however it will cost far less time once the following pull request is merged: 
https://review.openstack.org/#/c/52490/)

If you look at run_tests.py, you'll see that on line 104 it accepts a trove 
config file. If the run_tests.py script is updated to allow this value to be 
specified optionally via the command line, you could create a variation on 
"etc/trove/trove.conf.test" which specifies MongoDB. You'd then invoke 
run_tests.py with a "--group=" argument to run some subset of the tests support 
by the current Mongo DB code in fake mode.

Of course, this will do nothing to test the guest agent changes or confirm that 
the end to end system actually works, but it could help test a lot of 
incidental API and infrastructure database code.

As for real mode tests, I think we should wait until the datastore type / 
version code is finished, at which point I know we'll all be eager to add 
additional tests for these new datastores. Of course in the short term it 
should be possible for you to change the code locally to build a Mongo DB image 
as well as a Trove config file to support this and then just run some subset of 
tests that works with Mongo.

Thanks,

Tim



From: Illia Khudoshyn [ikhudos...@mirantis.com]
Sent: Monday, October 21, 2013 9:42 AM
To: OpenStack Development Mailing List
Subject: [openstack-dev] [Trove] Testing of new service types support

Hi all,

I've done with implementing the very first bits of MongoDB support in Trove 
along with unit tests and faced an issue with proper testing of it.

It is well known that right now only one service type per installation is 
supported by Trove (it is set in config). All testing infrastructure, including 
Trove-integration codebase and jenkins jobs, seem to rely on that service type 
as well. So it seems to be impossible to run all existing tests AND some 
additional tests for MongoDB service type in one pass, at least until Trove 
client will allow to pass service type (I know that there is ongoing work in 
this area).

Please note, that all of the above is about functional and intergation testing 
-- there is no issues with unit tests.

So the question is, should I first submit the code to Trove and then proceed 
with updating Trove-integration or just put aside all that MongoDB stuff until 
client (and -integration) will be ready?

PS AFAIK, there is some work on adding Cassandra and Riak (or Redis?) support 
to Trove. These guys will likely face this issue as well.

--

Best regards,

Illia Khudoshyn,
Software Engineer, Mirantis, Inc.



38, Lenina ave. Kharkov, Ukraine

www.mirantis.com

www.mirantis.ru



Skype: gluke_work

ikhudos...@mirantis.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Trove] How users should specify a datastore type when creating an instance

[Tim Simpson]

Thanks for the feedback Andrey.

>> 2. Got this case in irc, and decided to pass type and version together to 
>> avoid confusing.
I don't understand how allowing the user to only pass the version would confuse 
anyone. Could you elaborate?

>> 3. Names of types and maybe versions can be good, but in irc conversation 
>> rejected this case, i cant remember exactly reason.
Hmm. Does anyone remember the reason for this?

>> 4. Actually, "active" field in version marks it as default in type.
>> Specify default version in config can be usefull if you have more then one 
>> active versions in default type.
If 'active' is allowed to be set for multiple rows of the 'datastore_versions' 
table then it isn't a good substitute for the functionality I'm seeking, which 
is to allow operators to specify a *single* default version for each 
datastore_type in the database. I still think we should still add a 
'default_version_id' field to the 'datastore_types' table.

Thanks,

Tim


From: Andrey Shestakov [ashesta...@mirantis.com]
Sent: Monday, October 21, 2013 7:15 AM
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] [Trove] How users should specify a datastore type 
when creating an instance

1. Good point
2. Got this case in irc, and decided to pass type and version together to avoid 
confusing.
3. Names of types and maybe versions can be good, but in irc conversation 
rejected this case, i cant remember exactly reason.
4. Actually, "active" field in version marks it as default in type.
Specify default version in config can be usefull if you have more then one 
active versions in default type.
But how match active version in type depends on operator`s configuration. And 
what if "default version in config" will marked as inactive?

On 10/18/2013 10:30 PM, Tim Simpson wrote:
Hello fellow Trovians,

There has been some good work recently to figure out a way to specify a 
specific datastore  when using Trove. This is essential to supporting multiple 
datastores from the same install of Trove.

I have an issue with some elements of the proposed solution though, so I 
decided I'd start a thread here so we could talk about it.

As a quick refresher, here is the blue print for this work (there are some 
gists ammended to the end but I figured the mailing list would be an easier 
venue for discussion):
https://wiki.openstack.org/wiki/Trove/trove-versions-types

One issue I have is with the way the instance create call will change to 
support different data stores. For example, here is the post call:

"""
{
  "instance" : {
  "flavorRef" : "2",
  "name" : "as",
  "datastore_type" : "e60153d4-8ac4-414a-ad58-fe2e0035704a",
  "datastore_version" : "94ed1f9f-6c1a-4d6e-87e9-04ecff37b64b",
  "volume" : { "size" : "1" }
}
}
"""

1. I think since we have two fields in the instance object we should make a new 
object for datastore and avoid the name prefixing, like this:

"""
{
 "instance" : {
  "flavorRef" : "2",
  "name" : "as",
  "datastore": {
"type" : "e60153d4-8ac4-414a-ad58-fe2e0035704a",
"version" : "94ed1f9f-6c1a-4d6e-87e9-04ecff37b64b"
  }
  "volume" : { "size" : "1" }
}
}
"""

2. I also think a datastore_version alone should be sufficient since the 
associated datastore type will be implied:

"""
{
  "instance" : {
  "flavorRef" : "2",
  "name" : "as",
  "datastore": {
"version" : "94ed1f9f-6c1a-4d6e-87e9-04ecff37b64b"
  }
  "volume" : { "size" : "1" }
}
}
"""

3. Additionally, while a datastore_type should have an ID in the Trove 
infastructure database, it should also be possible to pass just the name of the 
datastore type to the instance call, such as "mysql" or "mongo". Maybe we could 
allow this in addition to the ID? I think this form should actually use the 
argument "type", and the id should then be passed as "type_id" instead.

"""
{
  "instance" : {
  "flavorRef" : "2",
  "name" : "as",
  "datastore": {
"type" : "mysql",
"version" : "94ed1f9f-6c1a-4d6e-87e9-04ecff37b64b"
  }
  "volume" : { "size" : "1" }
}
}

"""

Re: [openstack-dev] [Trove] How users should specify a datastore type when creating an instance

[Tim Simpson]

Hi Josh,

>> Given that Trove currently only supports a single datastore deployment per 
>> control system, does the current work also allow for a default type/version 
>> to be defined so that operators of Trove can set this as a property to 
>> maintain the current API compatibility/behavior?

Yes, the current pull request to support this allows for a default type, which, 
if there is only a single version for that type in the Trove infrastructure 
database, means that the existing behavior would be preserved. However as soon 
as an operator adds more than one datastore version of the default type then 
API users would need to always include the version ID. This would be fixed by 
recommendation #4 in my original message.

Thanks,

Tim



From: Josh Odom [josh.o...@rackspace.com]
Sent: Friday, October 18, 2013 3:16 PM
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] [Trove] How users should specify a datastore type 
when creating an instance

Hi Tim,
I do think your recommendation in 3 & 4 makes a lot of sense and improves the 
usability of the API.  Given that Trove currently only supports a single 
datastore deployment per control system, does the current work also allow for a 
default type/version to be defined so that operators of Trove can set this as a 
property to maintain the current API compatibility/behavior?

Josh


From: Tim Simpson mailto:tim.simp...@rackspace.com>>
Reply-To: OpenStack Development Mailing List 
mailto:openstack-dev@lists.openstack.org>>
Date: Friday, October 18, 2013 2:30 PM
To: 
"openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>" 
mailto:openstack-dev@lists.openstack.org>>
Subject: [openstack-dev] [Trove] How users should specify a datastore type when 
creating an instance

Hello fellow Trovians,

There has been some good work recently to figure out a way to specify a 
specific datastore  when using Trove. This is essential to supporting multiple 
datastores from the same install of Trove.

I have an issue with some elements of the proposed solution though, so I 
decided I'd start a thread here so we could talk about it.

As a quick refresher, here is the blue print for this work (there are some 
gists ammended to the end but I figured the mailing list would be an easier 
venue for discussion):
https://wiki.openstack.org/wiki/Trove/trove-versions-types

One issue I have is with the way the instance create call will change to 
support different data stores. For example, here is the post call:

"""
{
  "instance" : {
  "flavorRef" : "2",
  "name" : "as",
  "datastore_type" : "e60153d4-8ac4-414a-ad58-fe2e0035704a",
  "datastore_version" : "94ed1f9f-6c1a-4d6e-87e9-04ecff37b64b",
  "volume" : { "size" : "1" }
}
}
"""

1. I think since we have two fields in the instance object we should make a new 
object for datastore and avoid the name prefixing, like this:

"""
{
 "instance" : {
  "flavorRef" : "2",
  "name" : "as",
  "datastore": {
"type" : "e60153d4-8ac4-414a-ad58-fe2e0035704a",
"version" : "94ed1f9f-6c1a-4d6e-87e9-04ecff37b64b"
  }
  "volume" : { "size" : "1" }
}
}
"""

2. I also think a datastore_version alone should be sufficient since the 
associated datastore type will be implied:

"""
{
  "instance" : {
  "flavorRef" : "2",
  "name" : "as",
  "datastore": {
"version" : "94ed1f9f-6c1a-4d6e-87e9-04ecff37b64b"
  }
  "volume" : { "size" : "1" }
}
}
"""

3. Additionally, while a datastore_type should have an ID in the Trove 
infastructure database, it should also be possible to pass just the name of the 
datastore type to the instance call, such as "mysql" or "mongo". Maybe we could 
allow this in addition to the ID? I think this form should actually use the 
argument "type", and the id should then be passed as "type_id" instead.

"""
{
  "instance" : {
  "flavorRef" : "2",
  "name" : "as",
  "datastore": {
"type" : "mysql",
"version" : "94ed1f9f-6c1a-4d6e-87e9-04ecff37b64b"
  }
  "volume" : { "size" : "1" }
}
}

"""

4. Additionally, in the current pull request to implement this it is possible 
to avoid passing a version, but only if no more than one version of the 
datastore_type exists in the database.

I think instead the datastore_type row in the database should also have a 
"default_version_id" property, that an operator could update to the most recent 
version or whatever other criteria they wish to use, meaning the call could 
become this simple:

"""
{
  "instance" : {
  "flavorRef" : "2",
  "name" : "as",
  "datastore": {
"type" : "mysql"
  }
  "volume" : { "size" : "1" }
}
}
"""

Thoughts?

Thanks,

Tim
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Trove] How users should specify a datastore type when creating an instance

[Tim Simpson]

Hello fellow Trovians,

There has been some good work recently to figure out a way to specify a 
specific datastore  when using Trove. This is essential to supporting multiple 
datastores from the same install of Trove.

I have an issue with some elements of the proposed solution though, so I 
decided I'd start a thread here so we could talk about it.

As a quick refresher, here is the blue print for this work (there are some 
gists ammended to the end but I figured the mailing list would be an easier 
venue for discussion):
https://wiki.openstack.org/wiki/Trove/trove-versions-types

One issue I have is with the way the instance create call will change to 
support different data stores. For example, here is the post call:

"""
{
  "instance" : {
  "flavorRef" : "2",
  "name" : "as",
  "datastore_type" : "e60153d4-8ac4-414a-ad58-fe2e0035704a",
  "datastore_version" : "94ed1f9f-6c1a-4d6e-87e9-04ecff37b64b",
  "volume" : { "size" : "1" }
}
}
"""

1. I think since we have two fields in the instance object we should make a new 
object for datastore and avoid the name prefixing, like this:

"""
{
 "instance" : {
  "flavorRef" : "2",
  "name" : "as",
  "datastore": {
"type" : "e60153d4-8ac4-414a-ad58-fe2e0035704a",
"version" : "94ed1f9f-6c1a-4d6e-87e9-04ecff37b64b"
  }
  "volume" : { "size" : "1" }
}
}
"""

2. I also think a datastore_version alone should be sufficient since the 
associated datastore type will be implied:

"""
{
  "instance" : {
  "flavorRef" : "2",
  "name" : "as",
  "datastore": {
"version" : "94ed1f9f-6c1a-4d6e-87e9-04ecff37b64b"
  }
  "volume" : { "size" : "1" }
}
}
"""

3. Additionally, while a datastore_type should have an ID in the Trove 
infastructure database, it should also be possible to pass just the name of the 
datastore type to the instance call, such as "mysql" or "mongo". Maybe we could 
allow this in addition to the ID? I think this form should actually use the 
argument "type", and the id should then be passed as "type_id" instead.

"""
{
  "instance" : {
  "flavorRef" : "2",
  "name" : "as",
  "datastore": {
"type" : "mysql",
"version" : "94ed1f9f-6c1a-4d6e-87e9-04ecff37b64b"
  }
  "volume" : { "size" : "1" }
}
}

"""

4. Additionally, in the current pull request to implement this it is possible 
to avoid passing a version, but only if no more than one version of the 
datastore_type exists in the database.

I think instead the datastore_type row in the database should also have a 
"default_version_id" property, that an operator could update to the most recent 
version or whatever other criteria they wish to use, meaning the call could 
become this simple:

"""
{
  "instance" : {
  "flavorRef" : "2",
  "name" : "as",
  "datastore": {
"type" : "mysql"
  }
  "volume" : { "size" : "1" }
}
}
"""

Thoughts?

Thanks,

Tim
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TROVE] Thoughts on DNS refactoring, Designate integration.

[Tim Simpson]

Ilya you make a good point. We shouldn't spend time massively change the DNS 
code if we'll just have to do it again so that HEAT can do everything for us.

I echo Mike's comments though that if for some reason someone wants Designate 
support before we get HEAT integrated they should be able to add a new DNS 
driver. As I said before though I think that should be possible without major 
changes to the existing DNS code. 

Thanks,

Tim
___
From: Michael Basnight [mailto:mbasni...@gmail.com] 
Sent: Tuesday, October 01, 2013 5:37 PM
To: OpenStack Development Mailing List
Cc: OpenStack Development Mailing List
Subject: Re: [openstack-dev] [TROVE] Thoughts on DNS refactoring, Designate 
integration.

On Oct 1, 2013, at 3:06 PM, Ilya Sviridov  wrote:

On Tue, Oct 1, 2013 at 6:45 PM, Tim Simpson  wrote:
Hi fellow Trove devs,

With the Designate project ramping up, its time to refactor the ancient DNS 
code that's in Trove to work with Designate.

The good news is since the beginning, it has been possible to add new drivers 
for DNS in order to use different services. Right now we only have a driver for 
the Rackspace DNS API, but it should be possible to write one for Designate as 
well.

How it corelates with Trove dirrection to use HEAT for all provisioning and 
managing cloud resources? 
There are BPs for Designate resource 
(https://blueprints.launchpad.net/heat/+spec/designate-resource) and Rackspace 
DNS (https://blueprints.launchpad.net/heat/+spec/rax-dns-resource) as well and 
it looks logically to use the HEAT for that.
Currently Trove has logic for provisioning instances, dns driver, creation of 
security group, but with switching to HEAT way, we have duplication of the same 
functionality we have to support. 

+1 to using heat for this. However, as people are working on heat support right 
now to make it more sound, if there is a group that wants/needs DNS refactoring 
now, I'd say lets add it in. If no one is in need of changing what's existing 
until we get better heat support, then we should just abandon the review and 
leave the existing DNS code as is. 

I would prefer, if there is no one in need, to abandon the exiting review and 
add it to heat support. 


 

However, there's a bigger topic here. In a gist sent to me recently by Dennis 
M. with his thoughts on how this work should proceed, he included the comment 
that Trove should *only* support Designate: 
https://gist.github.com/crazymac/6705456/raw/2a16c7a249e73b3e42d98f5319db167f8d09abe0/gistfile1.txt

I disagree. I have been waiting for a canonical DNS solution such as Designate 
to enter the Openstack umbrella for years now, and am looking forward to having 
Trove consume it. However, changing all the code so that nothing else works is 
premature.

All non mainstream resources like cloud provider specific can be implemented as 
HEAT plugins (https://wiki.openstack.org/wiki/Heat/Plugins)
 

Instead, let's start work to play well with Designate now, using the open 
interface that has always existed. In the future after Designate enters 
integration status we can then make the code closed and only support Designate.

Do we really need playing with Designate and then replace it? I expect 
designate resource will come together with designate or even earlier.

With best regards,
Ilya Sviridov

Denis also had some other comments about the DNS code, such as not passing a 
single object as a parameter because it could be None. I think this is in 
reference to passing around a DNS entry which gets formed by the DNS instance 
entry factory. I see how someone might think this is brittle, but in actuality 
it has worked for several years so if anything changing it would introduce 
bugs. The interface was also written to use a full object in order to be 
flexible; a full object should make it easier to work with different types of 
DnsDriver implementations, as well as allowing more options to be set from the 
DnsInstanceEntryFactory. This later class creates a DnsEntry from an 
instance_id. It is possible that two deployments of Trove, even if they are 
using Designate, might opt for different DnsInstanceEntryFactory 
implementations in order to give the DNS entries associated to databases 
different patterns. If the DNS entry is created at this point its easier to 
further customize and tailor it. This will hold true even when Designate is 
ready to become the only DNS option we support (if such a thing is desirable).

Thanks,

Tim



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing lis

[openstack-dev] [TROVE] Thoughts on DNS refactoring, Designate integration.

[Tim Simpson]

Hi fellow Trove devs,

With the Designate project ramping up, its time to refactor the ancient DNS 
code that's in Trove to work with Designate.

The good news is since the beginning, it has been possible to add new drivers 
for DNS in order to use different services. Right now we only have a driver for 
the Rackspace DNS API, but it should be possible to write one for Designate as 
well.

However, there's a bigger topic here. In a gist sent to me recently by Dennis 
M. with his thoughts on how this work should proceed, he included the comment 
that Trove should *only* support Designate: 
https://gist.github.com/crazymac/6705456/raw/2a16c7a249e73b3e42d98f5319db167f8d09abe0/gistfile1.txt

I disagree. I have been waiting for a canonical DNS solution such as Designate 
to enter the Openstack umbrella for years now, and am looking forward to having 
Trove consume it. However, changing all the code so that nothing else works is 
premature.

Instead, let's start work to play well with Designate now, using the open 
interface that has always existed. In the future after Designate enters 
integration status we can then make the code closed and only support Designate.

Denis also had some other comments about the DNS code, such as not passing a 
single object as a parameter because it could be None. I think this is in 
reference to passing around a DNS entry which gets formed by the DNS instance 
entry factory. I see how someone might think this is brittle, but in actuality 
it has worked for several years so if anything changing it would introduce 
bugs. The interface was also written to use a full object in order to be 
flexible; a full object should make it easier to work with different types of 
DnsDriver implementations, as well as allowing more options to be set from the 
DnsInstanceEntryFactory. This later class creates a DnsEntry from an 
instance_id. It is possible that two deployments of Trove, even if they are 
using Designate, might opt for different DnsInstanceEntryFactory 
implementations in order to give the DNS entries associated to databases 
different patterns. If the DNS entry is created at this point its easier to 
further customize and tailor it. This will hold true even when Designate is 
ready to become the only DNS option we support (if such a thing is desirable).

Thanks,

Tim



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev