Yeah, I understand the timing aspect. If there's already a lot in progress, then I'd rather not step in front of that train. So as long as we agree going forward we look to guava or something, then we can figure out how to get there.
Regarding the storage framework, can Edison or Min chime in on how difficult/impactful it would be to move to synchronous? I could also help with it too. The longer we wait the more the impact if more storage drivers get contributed. Darren On Sep 5, 2013, at 4:49 PM, Kelven Yang <kelven.y...@citrix.com> wrote: > To tell the truth, I don't like the use of cglib in this case either, it > is per-request of making callback input available to IDE's intelli-sense > editing and be friendly to automatic code-refactoring through IDE > (Eclipse) which we do all the time. > > I'm fine with us to go with other standard approaches, my concern is the > timing. > > Kelven > > On 9/5/13 3:25 PM, "Darren Shepherd" <darren.s.sheph...@gmail.com> wrote: > >> Kelvin, >> >> I really have some concerns about the entire async framework that was put >> into place as part of the ipc framework. I don't know what VM sync is so >> I can't really comment too much on that. >> >> Basically my concern with the async framework is that it's yet another >> custom ACS thing. The use of cglib in the callbacks (or futures?) is to >> obscure and is not clear to a new person coming into ACS (for example, >> me). >> >> If we wish to do async stuff it should be based off of java 5 futures or >> guava's ListenableFutures. Those APIs and libraries are far more natural >> to java devs in the wild. Or we can get inspiration from scala and akka >> on how to do async well in java. In short, if we have a real use case >> for async, then I'd prefer we use an off the shelf library. >> >> In general I will lean towards most code being synchronous. To answer >> Chiradeeps comment on how do we do insane scale. That just requires true >> horizontal scalability of the management stack (which I'm not too sure if >> that's been fully proven beyond 3 servers and 30k hypervisors, but that's >> something we can address) and a really big database and somebody who >> enjoys really large failure domains. >> >> Darren >> >> On Sep 5, 2013, at 2:51 PM, Kelven Yang <kelven.y...@citrix.com> wrote: >> >>> New VM sync and job flow handling are moving towards event-driven model >>> which is async natural. Not sure about the details on how storage >>> framework is using async mechanism, but argument on callback to assume >>> memory state does not stand as strong as it might be, since arguments in >>> any regular call pass information in memory the same way. >>> >>> Excessive callbacks (async method) usage pattern does encourage people >>> to >>> program in a wrong direction of async programming, I would prefer to >>> substitute callback with event signaling and maintain/access >>> flow-context >>> through separated facility with well defined API, this can prevent >>> developers to pass too much information directly and create a >>> tight-coupling pattern. >>> >>> However, I'm a little concerning about yet another storage refactoring >>> right after its previous refactoring work is barely done. >>> >>> Kelven >>> >>> On 9/5/13 12:57 PM, "Chiradeep Vittal" <chiradeep.vit...@citrix.com> >>> wrote: >>> >>>> +1 for removing complexity especially if the sync pattern is being >>>> used on >>>> top of the async pattern. >>>> I see this behavior in the AgentManager.send as well -- even thought >>>> the >>>> AM >>>> is capable of async, practically nobody uses it as such. >>>> >>>> But I guess the question will arise : what if I do want more than 10^n >>>> long-running storage jobs >>>> (cos my cloud is as successful as AWS :)) >>>> >>>> On 9/4/13 5:03 PM, "Darren Shepherd" <darren.s.sheph...@gmail.com> >>>> wrote: >>>> >>>>> I've been reading over the storage code and have come to the >>>>> conclusion >>>>> that the async aspects of the storage framework should be removed. >>>>> >>>>> Whenever one introduces an async pattern you have to give a lot of >>>>> consideration to its use, benefits, and impact. Within the context of >>>>> ACS and given the current state of its code, I do not think it will be >>>>> possible to realize any benefits of the current callback approach. >>>>> Since nothing else in ACS uses callbacks, all of the async methods are >>>>> essentially wrapped in synchronous calls. So nothing as it stands is >>>>> actually async. >>>>> >>>>> Besides the current implementation, you need to conciser how you would >>>>> expect an implementation of the storage framework to use the callback. >>>>> The problem with callbacks is that they assume some in memory state. >>>>> This means if the process/server crashes that state is lost. Many >>>>> will >>>>> say just serialize the callback to the DB, but that is very >>>>> impractical. >>>>> >>>>> Since ACS doesn't actually stand in the data path, an async pattern >>>>> won't really even allow it to have better performance. ACS is just >>>>> waiting for some storage operation to happen. ACS can easily spawn >>>>> 1000 >>>>> threads and have them all wait. If you were to get to this point, >>>>> you'd >>>>> find that downstream you'll most likely have issues as you have 1000 >>>>> create template operations so its killing your filer. So you will >>>>> throttle storage operations to a level that won't kill your >>>>> infrastructure and that level is no where near the scalability limits >>>>> of >>>>> threads. >>>>> >>>>> The callbacks pattern really complicates the code and I see no real >>>>> benefit. Instead of spending a lot of effort trying to make all of >>>>> ACS >>>>> async to make it beneficial, I'd say that effort should be spent on >>>>> making ACS idempotent and crash-only. The point being, there's more >>>>> beneficial things we can do with our time. >>>>> >>>>> Given that only solidfire implements the new framework (and ACS legacy >>>>> too), I would assume its a simple things for Edison to just go and >>>>> quickly change it non-async. >>>>> >>>>> Darren >
