On 09/10/2012 08:15 PM, Ian Main wrote:
On Fri, Sep 07, 2012 at 02:35:39PM +0200, Jan Provaznik wrote:
On 09/04/2012 11:25 PM, Ian Main wrote:
On Mon, Sep 03, 2012 at 12:44:05PM +0200, Jan Provazník wrote:
On 08/30/2012 10:23 PM, Ian Main wrote:
On Thu, Aug 30, 2012 at 09:21:37AM +0200, Jan Provaznik wrote:
On 08/29/2012 09:55 PM, Ian Main wrote:
On Mon, Aug 27, 2012 at 02:47:42PM +0200, Jan Provaznik wrote:
On 08/21/2012 06:15 PM, Tomas Sedovic wrote:
[snip]
I think we would have to try it and see. Ultimately it all comes from a
database over a network connection. Querying heat only adds one more
http request/socket layer. Perhaps some time munging data but all in
all it's fairly light weight stuff.
Hi, sorry for late response, there was a deadline in last days...
Honestly I don't think that it would be light weight stuff, Heat
will have sooner or later some authentication mechanism, also it
will do another query (multiple queries) into its own DB backend.
As would Aeolus.. what I'm saying is the data and querying is similar in
both cases, there's just 1 extra http connection between heat and
aeolus. The trick would be getting the queries fine grained enough in
heat to allow aeolus to make single requests to get the info it needs.
(Warning: next paragraphs are boring and less important)
So if instances are saved in Conductor's DB then for serving a user
request is ideally required one DB query (you can join permissions and
other associations into one query), plus authentication stuff of course.
Now let's suppose that instances are kept on Heat side and Conductor
queries Heat directly. There are 2 options where to save instance's
metadata (attributes like pool, owner,...), based on chosen option, it
depends how the query would look like and number of required queries.
1) metadata are saved together with instance on Heat side (in the
similar way we did this for iwhd objects).
- then search query is done on Heat side, which implies that Heat's API
should support more complex searches, query then can looks like "give me
all instances in pool X owned by user Y"
- this means metadata mirroring between Conductor and Heat and makes
management of associated data more difficult (for example if I want to
delete/rename pool, I have to do proper update on Heat side too)
- you still have to do additional query on Conductor DB because of
permissions
So you would end up with following requests (minimally):
conductor db, heat api connection, heat's internal db connection
2) metadata are saved on Conductor side (together with instance
placeholder).
- whole search query is done on Conductor DB
- then you just query Heat with request "give me instances with ids
1,2,3,..."
So you would end up with same 3 requests too:
conductor db, heat api connection, heat's internal db connection
Both options looks like "far from ideal" solution to me.
The problem is that Instance is one of "essential" models in Conductor
and most of the Conductor stuff is related to this model. Trying to
extract it and make it external implies doing additional Heat queries
(these requests can't fully replace conductor's DB queries as shown
above) for many common user requests.
I think it's important to really consider what is involved in duplicating
As I mentioned in the first mail in this thread Conductor needs
primarily information about instance changes, from other resources
it checks also realms availability (maybe we might need check hw
profile changes in future).
that data. It has been difficult in the past eg with dbomatic
because it is a very hard problem. Even with events this doesn't
change. What happens when a service goes down for a time? You need to
replay events, read logs etc.. people spend a lot of time and a lot of
Well, for callback-related issues were discussed in a thread about
DC tracker and it wouldn't be so hard (it would be combination of
retry+replay).
This is what dbomatic had when it was using condor FYI.
Hm, IIRC dbomatic was checking for changes by reading condor log file
and kept logfile's position. After Conductor's failure and restart, it
continued from this saved position. I'm not sure how is this similar to
the callbacks discussed in DC Tracker thread.
What about this:
instead of callbacks we could use a messaging queue (an AMQP
implementation, e.g. RabbitMQ). Integrate this into Heat and
Conductor would be matter of few seconds (there are clients for most
of programming languages). This would be quite robust solution.
That would probably be a better idea. You will still need replay and
I'm not sure you can ever say you are getting 100% of the events, but I
think that is true with any solution.
The same messaging might be used also for sending notifications
between Conductor and Imagefactory.
money trying to solve data replication. IMO if you can access the
primary data source for your implementation that will be a far more
Don't forget that Heat is *not* primary data source (only cloud
provider is), And BTW, Heat will have to do the same "data
mirroring" (if you mean instance state checking by this term) itself
and if it's going to use dc-api you will probably end up with
polling providers.
Yes very true, but I what I am saying is why do it more than needed?
From what I've seen the real issue is actually ActiveRecord. The data
model is so tightly tied into the logic and application that using some
other data source becomes very difficult. I did some searching and I
think what I would propose is using something like the Repository
Pattern:
http://msdn.microsoft.com/en-us/library/ff649690.aspx
(Sorry it's an msdn page but it's the best description I could find.)
The idea would be to create a layer to allow for storage of data in
multiple places, including another web service. Perhaps this would be
useful for the "pluggable backend" idea?
Just a thought anyway, I'm probably in over my head now.. I'm no rails
pro :).
You are right that business logic is in model layer which is not optimal
(this is current Rails way). But this doesn't blocks you from using
other data sources - rails' model is already abstracted and decoupled
from activerecord. Also activeresource is probably what you are looking for:
https://github.com/rails/rails/tree/master/activemodel
https://github.com/rails/activeresource
Ian
Jan
