Ok, we'll keep on working on geting our project up and running using fey, and as soon as we need to implement fault-tolerance we'll get working on a design :)
On Tue, May 30, 2017 at 2:03 PM, Barbara Malta Gomes < [email protected]> wrote: > Hi Lucas, > > Glad to know that Fey suits your project. > > Why don't you guys start working on the clustering design and we can move > forward discussing about the details? > > I would like to see what you guys have in mind for Fey in "Cluster mode" > > Regards, > > On Mon, May 29, 2017 at 7:35 AM, Lucas Berri Cristofolini < > [email protected]> wrote: > >> Thank you for all the feedback you've given, Barbara and Tony! >> >> The more I play around with Fey, the more I think it suits our project >> very well, even if it isn't exactly IoT focused (although I suppose it does >> share some concepts). >> >> While the clustering for fault tolerance we were discussing isn't >> immediately necessary for us, it is on our roadmap and we find it essential >> for our project's completion. Having said that, we'd be absolutely willing >> to work together with you guys to bring that feature to fey. >> >> So how do you guys want to go about this? I don't know if Barbara has >> already started working on a design, if so we can discuss it with you guys >> and later help implement it. If not, maybe we could propose something >> ourselves and have the community discuss over it? >> >> Let me know what you think! >> >> On Fri, May 26, 2017 at 5:19 PM, Barbara Malta Gomes < >> [email protected]> wrote: >> >>> >>> Yes for both of your statement. >>> >>> I think it would be very valuable to start thinking and designing the >>> cluster implementation for Fey >>> >>> On Fri, May 26, 2017 at 12:18 PM, Rafael Weingärtner < >>> [email protected]> wrote: >>> >>>> Thank you again for the reply Barbara, >>>> >>>> Let’s see if I understood everything that we have discussed so far; Fey >>>> does only the recovery of agents that die within the same JVM (e.g. some >>>> unexpected exception that breaks the life cycle of an agent/performer). >>>> >>>> >>>> >>>> Your description of cluster is what I am looking for. To make sure I >>>> understood everything so far; the cluster feature you described does not >>>> exist yet, right? >>>> >>>> On Fri, May 26, 2017 at 2:14 PM, Barbara Malta Gomes < >>>> [email protected]> wrote: >>>> >>>>> Hi Rafael, >>>>> >>>>> The orchestrations running on Fey are fault-tolerant in the sense that >>>>> Fey makes sure that if the actor in the same JVM dies, it gets restarted a >>>>> certain number of times, but Fey has no implementation of fail-over, in >>>>> case the JVM crashes. >>>>> >>>>> About *what is a cluster in Fey Jargon*, in my view, I think a Fey >>>>> Cluster is a distributed system where we have Fey instances running in >>>>> different nodes (slaves) and we have a Fey Master node that is responsible >>>>> for the distribution and managing the orchestrations, not limiting the >>>>> scope of an orchestration to only one JVM. In that case we would need to >>>>> use Akka Remote actors because the performers of an orchestration could be >>>>> running in different nodes. In this context we would have a fail-over >>>>> mechanism in which the Fey master node would redistribute the performers >>>>> running on the JVM that crashed to other Fey nodes. >>>>> >>>>> Regards >>>>> >>>>> On Fri, May 26, 2017 at 8:07 AM, Rafael Weingärtner < >>>>> [email protected]> wrote: >>>>> >>>>>> Thanks for the reply Tony. From your answer, I am assuming that Fey >>>>>> does not have the ability to be fault tolerant. This means, using more >>>>>> than >>>>>> one runtime to host/execute agents, managing the fail-over in a >>>>>> transparent >>>>>> manner. >>>>>> >>>>>> Before we start discussing a feature suggestion, what is a cluster in >>>>>> Fey Jargon? >>>>>> >>>>>> Also, in Fey's Github README file I read the following: "Remote >>>>>> invocation of actors make it possible for Orchestrations running in one >>>>>> location to interact with Orchestrations running in another where >>>>>> services >>>>>> not available locally can be accessed". When I read that I felt that I >>>>>> could invoke remote agents (performers in different runtime) in a >>>>>> transparent manner. By transparent here I mean, sending a message to an >>>>>> agent using an identifier, and then the platform (Fey) takes care of >>>>>> locating the agent and actually delivering the message. >>>>>> >>>>>> On Thu, May 25, 2017 at 5:22 PM, Tony Faustini <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi Rafael, good question what is meant by a Fey cluster? The Fey >>>>>>> that I can download today runs at on single server (say a Raspberry Pi >>>>>>> at >>>>>>> the low end) and I would like also to run Fey at the Cluster level (say >>>>>>> on >>>>>>> a Mesos cluster). My use case of has Fey running standalone on >>>>>>> micro-servers that are connected to the cloud. At the cloud I am >>>>>>> running a >>>>>>> Mess cluster and I want to replace some existing technology running on >>>>>>> Mesos with a version of Fey that will run across that Mesos Cluster. >>>>>>> This >>>>>>> is one meaning of running Fey as a cluster that I would like to see. I >>>>>>> would like to see my performers with upwards and downwards scalability >>>>>>> over >>>>>>> a cluster. >>>>>>> >>>>>>> Just for clarification my use case doesn’t look at the micro-servers >>>>>>> and the cloud Mesos cluster as one large network on which I want to run >>>>>>> a >>>>>>> single instance of clustered Fey. It’s a simpler use case in which each >>>>>>> micros-server runs a standalone version of Fey and on my Mesos cluster I >>>>>>> need a version of Fey on the iota roadmap that will allow me to >>>>>>> distribute >>>>>>> my orchestrations/performers across my Mesos cluster. Since Akka is >>>>>>> part of >>>>>>> the SMACK stack (Spark, Mesos, Akka, Cassandra and Kafka) there may >>>>>>> already >>>>>>> be a way of running Fey on a Mesos cluster. I would like to see the Fey >>>>>>> cluster version on the iota dev roadmap. >>>>>>> >>>>>>> There might be some interesting use cases in which Akka remoting >>>>>>> might be used. I would love to hear about them and in general other use >>>>>>> cases that folks have in mind for Fey. >>>>>>> >>>>>>> -Tony >>>>>>> >>>>>>> >>>>>>> On May 25, 2017, at 11:58 AM, Rafael Weingärtner < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>> Thanks for the feedback Barbara. Lucas is in route today, so I am >>>>>>> answering on his behalf. >>>>>>> >>>>>>> I think our confusion is with this expression “cluster context”. >>>>>>> What do you (Fey) see as a cluster? Only a group of performers >>>>>>> (agents) in the same runtime (JVM)? >>>>>>> >>>>>>> Also, about these security issues, are you talking about the >>>>>>> security (confidentiality and integrity) of data transmitted from one >>>>>>> performer to the other remotely? >>>>>>> Regarding performers being unreachable; this might be the case if >>>>>>> they are inside a NAT. However, in the IoT space, the idea is to plug >>>>>>> everything directly to the Internet using IPv6; so, this should not be a >>>>>>> problem, right? At least that is how I see this expression; I do not >>>>>>> work >>>>>>> with IoT, so I am probably wrong here. >>>>>>> >>>>>>> Having said that, our necessity is not in the IoT space, but still, >>>>>>> Fey’s design fits quite well our system’s requirements. The only point >>>>>>> is >>>>>>> that we need our agents/performs to be fault-tolerant in case their >>>>>>> runtime >>>>>>> crashes. Therefore, redundancy for the performers (agents) platform is a >>>>>>> must. >>>>>>> >>>>>>> If this is interesting for Fey as a community we could start working >>>>>>> together on this matter. >>>>>>> >>>>>>> On Thu, May 25, 2017 at 1:25 PM, Barbara Malta Gomes < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi Lucas, >>>>>>>> >>>>>>>> Unfortunately Fey does not support remote connection between >>>>>>>> performers. >>>>>>>> >>>>>>>> Fey is not meant to be a connectivity protocol like MQTT or other >>>>>>>> messaging framework. >>>>>>>> The reason we do not implemented remote actors outside of a cluster >>>>>>>> context is because of security and also in most use cases the >>>>>>>> Performers >>>>>>>> will be unreachable from the "outside" (in a intranet). >>>>>>>> >>>>>>>> What we have today is an MQTT publisher Performer on one side and a >>>>>>>> MQTT subscriber in the other side. >>>>>>>> >>>>>>>> But we can definitely discuss more about implementing this >>>>>>>> functionality on Fey. >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> >>>>>>>> On Wed, May 24, 2017 at 12:55 PM, Lucas Berri Cristofolini < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Both, actually... >>>>>>>>> >>>>>>>>> I suppose clusterization for fault tolerance is actually a more >>>>>>>>> immediate concern, but the ability to run performers remotely is very >>>>>>>>> interesting as well :) >>>>>>>>> >>>>>>>>> On May 24, 2017 4:47 PM, "Rafael Weingärtner" < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Do you want only peer-peer communication? Or do you want >>>>>>>>>> clustering of agents for fault-tolerant systems? >>>>>>>>>> http://doc.akka.io/docs/akka/snapshot/java/common/cluster.html >>>>>>>>>> >>>>>>>>>> On Wed, May 24, 2017 at 3:27 PM, Lucas Berri Cristofolini < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Yup, that worked just fine! >>>>>>>>>>> >>>>>>>>>>> I wrote the example you showed at ApcheCon in Java to test it >>>>>>>>>>> out, and I was now wondering if it is possible to run the two >>>>>>>>>>> performers in >>>>>>>>>>> that ensemble in separate hosts. I suppose this would be akin to >>>>>>>>>>> Akka's >>>>>>>>>>> 'remoting' (http://doc.akka.io/docs/akka/ >>>>>>>>>>> snapshot/java/remoting.html). >>>>>>>>>>> >>>>>>>>>>> I saw some of the examples in GitHub but it isn't quite clear to >>>>>>>>>>> me what I need to set in order for the performers to be able to >>>>>>>>>>> communicate >>>>>>>>>>> in that scenario. Are there any settings I can specify in the >>>>>>>>>>> orchestration >>>>>>>>>>> json to point to the remote host that would be running the other >>>>>>>>>>> performer? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> PS: Thanks for the quick reply yesterday! >>>>>>>>>>> >>>>>>>>>>> On Tue, May 23, 2017 at 3:35 PM, Barbara Malta Gomes < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Lucas, >>>>>>>>>>>> >>>>>>>>>>>> Good to hear that you are trying out iota. >>>>>>>>>>>> >>>>>>>>>>>> Sure, you can definitely write the performers in java. >>>>>>>>>>>> You just have to do like you said, add the Fey .jar as a >>>>>>>>>>>> dependency, implement a Performer that inherits from the >>>>>>>>>>>> FeyGenericActor >>>>>>>>>>>> and override the functions. >>>>>>>>>>>> >>>>>>>>>>>> Once you have done that, generate the .jar for the performer >>>>>>>>>>>> and use it on Fey. >>>>>>>>>>>> >>>>>>>>>>>> Let us know if you have any problems. >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 23, 2017 at 11:27 AM, Lucas Berri Cristofolini < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hello, Iota community! :) >>>>>>>>>>>>> >>>>>>>>>>>>> One of my colleagues watched Barbara's talk at ApacheCon 2017 >>>>>>>>>>>>> in Miami, and we were really interested in the way the Fey Engine >>>>>>>>>>>>> works. >>>>>>>>>>>>> >>>>>>>>>>>>> I've been wanting to try it out since then but only got around >>>>>>>>>>>>> to it today, and after going through the GitHub page for the >>>>>>>>>>>>> project and >>>>>>>>>>>>> Barbara's slides, I'm wondering if it is possible to write the >>>>>>>>>>>>> performers >>>>>>>>>>>>> in Java, since the Engine seems to take compiled .jars. >>>>>>>>>>>>> >>>>>>>>>>>>> If that's possible, how would one go about doing it? Is it >>>>>>>>>>>>> just a matter of adding the fey engine as a dependency and >>>>>>>>>>>>> overriding the >>>>>>>>>>>>> right functions? >>>>>>>>>>>>> >>>>>>>>>>>>> Cheers! >>>>>>>>>>>>> Lucas >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Barbara Gomes >>>>>>>>>>>> Computer Engineer >>>>>>>>>>>> San Jose, CA >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Rafael Weingärtner >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Barbara Gomes >>>>>>>> Computer Engineer >>>>>>>> San Jose, CA >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Rafael Weingärtner >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Rafael Weingärtner >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Barbara Gomes >>>>> Computer Engineer >>>>> San Jose, CA >>>>> >>>> >>>> >>>> >>>> -- >>>> Rafael Weingärtner >>>> >>> >>> >>> >>> -- >>> Barbara Gomes >>> Computer Engineer >>> San Jose, CA >>> >> >> > > > -- > Barbara Gomes > Computer Engineer > San Jose, CA >
