Hi Nan, In my understanding, the design of piper supports exactly the case you're saying. You can have a producer app that produces a stream and consumer apps subscribe to it. My point was simply that you need a third app to be the producer of the stream. The consumers of the stream do not need to be aware of how the stream is being produced.
-Flavio On May 28, 2012, at 12:50 PM, Nan N. Jiang wrote: > Hi Flavio, > > I think in the ideal case, S4 should act as a real cluster. > We can deploy our APP in the S4 cluster and we don't need to think about how > and where the app is deployed. > The APP just need to specify which stream(event type) it's interested, then > if there is event generated for this stream, the event will flow to this > APP. S4 as a platform should help to handle this instead of using feeds > copies. > > With this feature, APP will be free from finding the way how to get the > incoming events and focus what kind of events it want. > > -Nan > > > > On 5/25/12 1:50 PM, "Flavio Junqueira" <f...@yahoo-inc.com> wrote: > >> Nan, I'm not sure I understand what sharing means. Does it mean that each app >> gets a copy of the stream or you want each app to get a fraction of the >> stream? In any case, you seem to need a third app perhaps that feeds copies >> of >> the stream. >> >> -Flavio >> >> On May 11, 2012, at 11:58 AM, Nan N. Jiang wrote: >> >>> Yes. If there is no sharing or dependencies between apps, there is no need >>> to share the cluster. >>> >>> But if there is sharing or dependencies between apps, for example: sharing >>> event stream(thinking about more then one apps are interested in the same >>> event stream case), how could s4 solve this issue with logical cluster? >>> >>> -- Nan >>> >>> On 5/8/12 9:30 PM, "Leo Neumeyer" <leoneume...@gmail.com> wrote: >>> >>>> Seems to me that the simplest solution is to create a logical cluster for >>>> each app or group of apps that need to be isolated. Using various smaller >>>> clusters may be more efficient that using a single large one because there >>>> fewer nodes use less network traffic and makes it easier to plan resource >>>> allocation. If there is no sharing or dependencies between apps, there is >>>> no need to share the cluster. The tools will make it easy to create >>>> clusters. Does this make sense for your use case? >>>> >>>> -leo >>>> >>>> On Tue, May 1, 2012 at 12:08 PM, Matthieu Morel <mmo...@apache.org> wrote: >>>> >>>>> On 4/29/12 2:49 AM, Nan N. Jiang wrote: >>>>> >>>>>> Hi Team, >>>>>> >>>>>> I want to start a discuss about ŒRuntime isolation¹ for s4. >>>>>> >>>>>> Currently, in S4 and S4-piper, following S4¹s symmetric principal, all >>>>>> applications will deployed in all nodes. >>>>>> And when starting s4 service, all applications deployed will run in the >>>>>> same JVM. >>>>>> This can be a disaster. >>>>>> Different application will fight for the JVM/hardware resource. >>>>>> Bad application even could kill other applications. >>>>>> >>>>>> Yes. Application Isolation could release the pain. >>>>>> But all applications are still fight for the resource in the same node. >>>>>> And all the PE in the same Application still fight for JVM resource. >>>>>> >>>>>> Thinking about a big application which contains hundreds of PE >>>>>> prototypes, >>>>>> If we want start this application in one JVM, >>>>>> Our s4 cluster must have a set of very high performance machines due to >>>>>> the CPU/Memory and other requirement of this application. >>>>>> This will limit the usage of S4 in big solution. >>>>>> I think the future of s4 cluster will be a big set of normal boxes >>>>>> instead of a small set of high performance boxes. >>>>>> >>>>>> So, compare application isolation, could we go further? >>>>>> Could we provide Runtime Isolation function for S4? >>>>>> Could we change the local deploy unit from application to a group of PE? >>>>>> even one PE? >>>>>> If this function can be provided by s4, >>>>>> Every PE could run in it¹s own JVM if needed. >>>>>> Then the resource fighting between PE will be minimum. >>>>>> >>>>> >>>>> Deploying different PEs to different nodes would break the symmetry which >>>>> is a key property of the design of S4. However, you are pointing out a >>>>> real >>>>> issue, and one way to provide PE isolation might be at the level of >>>>> partitioning. We could provide a way to provide exclusive partitioning for >>>>> some PEs using a partition exclusiveness scheme, which would be propagated >>>>> to every node. Concretely, messages for PE with key K (or a set of keys) >>>>> would be sent to node N, and only messages with such keys would be sent to >>>>> node N. This way, we provide PE isolation, without modifying the design of >>>>> the platform. >>>>> >>>>> Do you think this could properly address the isolation issue? >>>>> To all: would that harm the overall design of the platform? >>>>> >>>>> Thanks! >>>>> >>>>> Matthieu >>>>> >>>>> >>>>> >>>>>> Let¹s discuss about this. >>>>>> >>>>>> Regards, >>>>>> Jiang Nan >>>>>> >>>>>> >>>>> >>>> >>> >> >> flavio >> junqueira >> senior research scientist >> >> f...@yahoo-inc.com >> direct +34 93-183-8828 >> >> avinguda diagonal 177, 8th floor, barcelona, 08018, es >> phone (408) 349 3300 fax (408) 349 3301 >> > flavio junqueira senior research scientist f...@yahoo-inc.com direct +34 93-183-8828 avinguda diagonal 177, 8th floor, barcelona, 08018, es phone (408) 349 3300 fax (408) 349 3301
