My initial hunch is that these are different goals and therefore concur with your thought that you could do this in phase 2.
On Mon, Oct 30, 2017 at 3:03 PM, Saikat Kanjilal <sxk1...@hotmail.com> wrote: > The goal is to tie together spark streaming and heron, I could very easily > see an architecture where we build topologies in heron to apply business > logic and pump them into time windows with spark streaming where can apply > machine learning algorithms inside these windows. > > > Detailed Example: > > > Stock Trading or gamin use case: > > > Heron is used to extract real time data from a gaming or trading client, > > Heron will apply a set of business rules to cleanup this data to fit into > a set of machine learning models > > We need to then figure out how to connect spark streaming to heron, in the > case of spark streaming it listens to some tcp or http endpoint so the > objective would be to figure out the bridge between the two worlds. > > > My point here is that if I am going through and building out a scala API > it might make sense to also build a connector to a tech stack entirely > written using scala as a first class citizen. > > > > Let me know your thoughts, we can also push this to phase 2 once the heron > scala api is readily available. > > > ________________________________ > From: Sanjeev Kulkarni <sanjee...@gmail.com> > Sent: Monday, October 30, 2017 2:54 PM > To: dev@heron.incubator.apache.org > Subject: Re: Hello heron community > > I didn't really get what you meant by 'stream endpoint would be coming from > heron'. Could you please elaborate? Preferably with an example? > > On Mon, Oct 30, 2017 at 2:26 PM, Saikat Kanjilal <sxk1...@hotmail.com> > wrote: > > > I'm conflicted a bit on 1, here's why I feel like most of our users would > > want to use the spouts/bolts API who are coming over from Storm or using > > heron for the first time, however the Streamlets interface seems like it > > has the right level of abstraction, I think I will start with the > Streamlet > > interface and add in the spout/bolt API as needed. One other thought I > > had was to figure out an integration plan with spark, to this end the fit > > between heron and spark would be in using spark streaming where the > stream > > endpoint would be coming from heron, not sure if that should be part of > > this effort or not, what do you guys think? > > > > > > ________________________________ > > From: Sanjeev Kulkarni <sanjee...@gmail.com> > > Sent: Monday, October 30, 2017 2:17 PM > > To: dev@heron.incubator.apache.org > > Subject: Re: Hello heron community > > > > I have a couple of questions > > 1. Do you plan on exposing both the low level(spouts/bolts) and the > > streamlet api? Or are you preferring one over another? > > 2. My suggestion would be to start with the Streamlet interface. > Primarily > > because a) Most of scala ml libraries that I've seen tend to operate on > the > > streamlet kind of interface and b) It has a far smaller surface area(i.e. > > number of interfaces) so might be easy to get something up quickly for > > testing. > > Thanks! > > > > On Mon, Oct 30, 2017 at 2:13 PM, Saikat Kanjilal <sxk1...@hotmail.com> > > wrote: > > > > > Thanks Sanjeev, I had an initial idea that I wanted to float on the > > list, > > > I was thinking that as part of the initial scala port of the API I'd > like > > > to propose that we use this tool: http://javatoscala.com/ > Java to Scala converter<http://javatoscala.com/> > javatoscala.com > How does it work? Java to Scala converter is created with Play framework > and Scalagen library. I don't want you to see my code. No problem. Java to > Scala converter ... > > > > > Java to Scala converter<http://javatoscala.com/> > Java to Scala converter<http://javatoscala.com/> > javatoscala.com > How does it work? Java to Scala converter is created with Play framework > and Scalagen library. I don't want you to see my code. No problem. Java to > Scala converter ... > > > > > javatoscala.com > > How does it work? Java to Scala converter is created with Play framework > > and Scalagen library. I don't want you to see my code. No problem. Java > to > > Scala converter ... > > > > > > > > > > > > Java to Scala converter<http://javatoscala.com/> > Java to Scala converter<http://javatoscala.com/> > javatoscala.com > How does it work? Java to Scala converter is created with Play framework > and Scalagen library. I don't want you to see my code. No problem. Java to > Scala converter ... > > > > > Java to Scala converter<http://javatoscala.com/> > Java to Scala converter<http://javatoscala.com/> > javatoscala.com > How does it work? Java to Scala converter is created with Play framework > and Scalagen library. I don't want you to see my code. No problem. Java to > Scala converter ... > > > > > javatoscala.com > > How does it work? Java to Scala converter is created with Play framework > > and Scalagen library. I don't want you to see my code. No problem. Java > to > > Scala converter ... > > > > > > > > > javatoscala.com > > > How does it work? Java to Scala converter is created with Play > framework > > > and Scalagen library. I don't want you to see my code. No problem. Java > > to > > > Scala converter ... > > > > > > > > > I propose that we use the above tool to do an initial conversion of all > > > the Java functional and oo interfaces and then plug that into the > initial > > > design doc as well as the codebase to get things off the ground. I > have > > > already forked the heron repo and will be doing this off of a child > > branch > > > based on my fork. > > > > > > > > > How does that sound to folks, anyone has strong objections, of course I > > > fully realize that I will still need to do a lot of work around > > refactoring > > > the code even when using the converter so I'm prepared for that, the > goal > > > in using the converters is to save a bit of time so that manual > > > intervention isnt necessarily needed in designing every interface. > > > > > > Thoughts. > > > > > > ________________________________ > > > From: Sanjeev Kulkarni <sanjee...@gmail.com> > > > Sent: Sunday, October 29, 2017 10:18 PM > > > To: dev@heron.incubator.apache.org > > > Subject: Re: Hello heron community > > > > > > Hi Saikat, > > > Welcome to the Heron users group. Your plan of action sounds good to > me. > > A > > > native scala interface would be great for all those scala fans. Eagerly > > > awaiting design doc. > > > Thanks! > > > > > > On Sun, Oct 29, 2017 at 6:17 PM, Saikat Kanjilal <sxk1...@hotmail.com> > > > wrote: > > > > > > > Hello Folks, > > > > > > > > I'm interested in taking and driving the following github component > > (this > > > > means design/development/architecture and more). > > > > > > > > > > > > https://github.com/twitter/heron/issues/668 > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https:// > github.com/twitter/heron/issues/668> > > Ability to write native scala topologies · Issue #668 · twitter/heron< > https://github.com/twitter/heron/issues/668> > github.com > Support for Scala Instance and APIs. > > > > > > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https:// > > > github.com/twitter/heron/issues/668> > > > > > > Ability to write native scala topologies · Issue #668 · twitter/heron< > > > https://github.com/twitter/heron/issues/668> > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https:// > github.com/twitter/heron/issues/668> > > Ability to write native scala topologies · Issue #668 · twitter/heron< > https://github.com/twitter/heron/issues/668> > github.com > Support for Scala Instance and APIs. > > > > > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https:// > > github.com/twitter/heron/issues/668> > > > > Ability to write native scala topologies · Issue #668 · twitter/heron< > > https://github.com/twitter/heron/issues/668> > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https:// > github.com/twitter/heron/issues/668> > > Ability to write native scala topologies · Issue #668 · twitter/heron< > https://github.com/twitter/heron/issues/668> > github.com > Support for Scala Instance and APIs. > > > > > github.com > > Support for Scala Instance and APIs. > > > > > > > > > github.com > > > Support for Scala Instance and APIs. > > > > > > > > > > > > > > > > > > > > > [https://avatars1.githubusercontent.com/u/367684?v=4&s=400]<https:// > > > > github.com/twitter/heron/issues/668> > > > > > > > > Ability to write native scala topologies · Issue #668 ...< > > > > https://github.com/twitter/heron/issues/668> > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https:// > github.com/twitter/heron/issues/668> > > Ability to write native scala topologies · Issue #668 · twitter/heron< > https://github.com/twitter/heron/issues/668> > github.com > Support for Scala Instance and APIs. > > > > > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https:// > > github.com/twitter/heron/issues/668> > > > > Ability to write native scala topologies · Issue #668 · twitter/heron< > > https://github.com/twitter/heron/issues/668> > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https:// > github.com/twitter/heron/issues/668> > > Ability to write native scala topologies · Issue #668 · twitter/heron< > https://github.com/twitter/heron/issues/668> > github.com > Support for Scala Instance and APIs. > > > > > github.com > > Support for Scala Instance and APIs. > > > > > > > > > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https:// > > > github.com/twitter/heron/issues/668> > > > > > > Ability to write native scala topologies · Issue #668 · twitter/heron< > > > https://github.com/twitter/heron/issues/668> > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https:// > github.com/twitter/heron/issues/668> > > Ability to write native scala topologies · Issue #668 · twitter/heron< > https://github.com/twitter/heron/issues/668> > github.com > Support for Scala Instance and APIs. > > > > > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https:// > > github.com/twitter/heron/issues/668> > > > > Ability to write native scala topologies · Issue #668 · twitter/heron< > > https://github.com/twitter/heron/issues/668> > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https:// > github.com/twitter/heron/issues/668> > > Ability to write native scala topologies · Issue #668 · twitter/heron< > https://github.com/twitter/heron/issues/668> > github.com > Support for Scala Instance and APIs. > > > > > github.com > > Support for Scala Instance and APIs. > > > > > > > > > github.com > > > Support for Scala Instance and APIs. > > > > > > > > > > > > > github.com > > > > heron - Heron is a realtime, distributed, fault-tolerant stream > > > processing > > > > engine from Twitter > > > > > > > > > > > > > > > > > > > > I've joined the heron slack channel as recommended and am ready to > > start > > > > by creating a master JIRA issue to track this if not already done, I > > > wanted > > > > some guidance from a committer or two to help guide me a bit take > care > > of > > > > at he administrative details and how to get this kicked off the > ground. > > > My > > > > background includes working on apache reef codebase as well as > dabbling > > > in > > > > apache mahout and apache flume as well as storm and various > nosql/graph > > > > databases. Looking forward to being part of the community. > > > > > > > > > > > > > > > > The rough plan of action in my mind so you know: > > > > > > > > 1) Review heron architecture and codebase in detail > > > > > > > > 2) Create an umbrella JIRA with the first series of tasks > > > > > > > > 3) Come up with a design doc > > > > > > > > 4) Build the interfaces > > > > > > > > 5) Add more dev tasks to JIRA > > > > > > > > 6) Drive the implementation > > > > > > > > > > > > Thanks in advance for your help. > > > > > > > > > > > > > >
