On 01/23/2013 11:51 PM, Steve Loranz wrote:
On Jan 23, 2013, at 4:21 PM, Matt Wagner <[email protected]>
wrote:
On Wed, Jan 23, 2013 at 01:42:31PM -0500, Mo Morsi wrote:
On 01/23/2013 01:16 PM, Bryan Kearney wrote:
Well, just thinking out loud.. can any component go into the
cloud which may not be in the same VPN space?
-- bk
I'd imagine this would come down to the security policy of the
organization deploying to the cloud, namely how lax the firewall
can be to permit connections from the cloud as well as the ip
address assignment on the cloud instances launched.
For a while now, I've been watching this discussion thinking, "It
feels like we decided a while back that AMQP would be cool, and
are now trying to work backwards to come up with reasons to
justify it." Perhaps I'm just not forward-looking enough, or
perhaps I'm overlooking an important detail, but that's sort of how
it feels to me.
I think networking is an edge case, and a minor detail. We should
absolutely try to support running components on different boxes
(though I don't believe we have ever properly tested or supported
it). But if
Having Aeolus components on different boxes is not edge case for me. I
think that having Imagefactory on separate box will be quite often
production setup requirement: if a production box is not bare metal and
doesn't support nested virtualization then separate box is the only
option. Also Imagefactroy HW requirements (more storage, less CPU) are
different from other Aeolus services (more CPU, less storage).
Personally I use remote Imagefactory or Deltacloud services quite often.
you break Aeolus up and run it on disparate network segments,
you're going to have to handle somehow patching things together. I
don't think we should choose how we handle inter-component
messaging based on what happens if you break components up on
different networks and refuse to set up a VPN or appropriate
port-forwarding rules.
Agree. I wouldn't care about cross VPN support.
I don't mean to single out networking in general, though; it's
just the latest in the discussion. I just worry that the discussion
is largely theoretical and academic, focused on how different
means of inter-component communications differ. That might be a
good conversation to have if we didn't already have all our
components using one. What I'm missing from this discussion is an
exploration of what issues we're actually experiencing today with
our HTTP callbacks system, and whether the overhead of switching to
AMQP is a worthwhile trade-off. Is changing Factory and Conductor
to use AMQP worthwhile to prevent the issue where if you shut down
one of the two components in the middle of an exchange of messages,
some messages might be lost? Could that better be solved by
implementing some queuing or polling? Or, is it fair to say that if
you send a job to Factory from Conductor and then shut down
Conductor, it's just expected that the updated status might be
missed?
There are various not-theoretical failures which can cause that a
callback is not delivered, off-hand examples:
- network/firewall error
- vpn is down
- callback receiver is down
- rails proxy (in our case apache) is down
My impression is that overall opinion is that a failure can occur so
sporadically that there is no reason to take care of them. Based on my
experience I believe this is wrong. I hit all above errors myself when
testing imagefactory (except the last apache proxy error, but only
because I was accessing rails directly).
Another opinion mentioned here and on IRC was that failures could be
covered by additional polling/status checking. This is little bit
suboptimal:
1) it's not sufficient to check objects states after a service is back
after a failure (callback failure can occur even if both ends are
running). This polling would have to be done repeatedly, as a receiver
you don't know if a callback delivery failed or if it wasn't sent yet
2) you can use just polling in such case and get rid of callbacks at all
3) you have two ways through which the object state is changed (through
callbacks and through polling)
4) you have to take care of potential race conditions between polling
and callbacks
5) polling is not efficient
And I want to emphasize that it's not a problem only of
Imagefactory<->Conductor communication but all components I mentioned in
the first mail.
6) means bunch of additional code/logic on receiver side
Can we agree on the assumption that current callback solution is not
sufficient and that more robustness is required?
IMO we have 2 options then:
1) make callback system more robust in *each* component. As I said
before: if general opinion is that this is preferred solution, I'm fine
with it. Though to me this sounds like reinventing of wheel to some extent.
2) use message bus instead of callbacks - this brings you all required
features out of the box, also I would expect this means less coding and
delegating the problem to third party system which is designed exactly
for this purpose, but the feedback so far was quite negative about this
option.
It's not my intention to vehemently oppose AMQP, and I certainly
don't mean to suggest that it shouldn't be discussed. I just don't
find the current conversation terribly productive at making the
case for why we should switch.
-- Matt
imagefactory started off using QMF. It's what we were told was
decided on when Aeolus was designed. But conductor was having a
difficult time using it because it meant having a separate thread or
process to bridge conductor to the broker. So, in the summer of
2011, there was a number of discussions where developers on both the
imagefactory and conductor teams came out in favor of imagefactory
offering a REST interface. We did, and near the end of January of
2012, we officially removed the QMF interface from the source tree
when it started to seem like QPID/QMF was failing to gain traction
in the wider community.
I'm not saying the conversation shouldn't happen either. What I am
saying is that it should pick up where it was left 18 months ago
with the question of if the challenges of conductor actually
connecting to a broker are easier to deal with now than they were in
2011 when they were great enough to decide to switch course.
Could someone please send me a link to the discussion (if the discussion
was on the mailing list) - I can't find it, so I can't pick up
conversation from that point.
-steve
Jan
