The goal is to tie together spark streaming and heron, I could very easily see an architecture where we build topologies in heron to apply business logic and pump them into time windows with spark streaming where can apply machine learning algorithms inside these windows.
Detailed Example: Stock Trading or gamin use case: Heron is used to extract real time data from a gaming or trading client, Heron will apply a set of business rules to cleanup this data to fit into a set of machine learning models We need to then figure out how to connect spark streaming to heron, in the case of spark streaming it listens to some tcp or http endpoint so the objective would be to figure out the bridge between the two worlds. My point here is that if I am going through and building out a scala API it might make sense to also build a connector to a tech stack entirely written using scala as a first class citizen. Let me know your thoughts, we can also push this to phase 2 once the heron scala api is readily available. ________________________________ From: Sanjeev Kulkarni <sanjee...@gmail.com> Sent: Monday, October 30, 2017 2:54 PM To: dev@heron.incubator.apache.org Subject: Re: Hello heron community I didn't really get what you meant by 'stream endpoint would be coming from heron'. Could you please elaborate? Preferably with an example? On Mon, Oct 30, 2017 at 2:26 PM, Saikat Kanjilal <sxk1...@hotmail.com> wrote: > I'm conflicted a bit on 1, here's why I feel like most of our users would > want to use the spouts/bolts API who are coming over from Storm or using > heron for the first time, however the Streamlets interface seems like it > has the right level of abstraction, I think I will start with the Streamlet > interface and add in the spout/bolt API as needed. One other thought I > had was to figure out an integration plan with spark, to this end the fit > between heron and spark would be in using spark streaming where the stream > endpoint would be coming from heron, not sure if that should be part of > this effort or not, what do you guys think? > > > ________________________________ > From: Sanjeev Kulkarni <sanjee...@gmail.com> > Sent: Monday, October 30, 2017 2:17 PM > To: dev@heron.incubator.apache.org > Subject: Re: Hello heron community > > I have a couple of questions > 1. Do you plan on exposing both the low level(spouts/bolts) and the > streamlet api? Or are you preferring one over another? > 2. My suggestion would be to start with the Streamlet interface. Primarily > because a) Most of scala ml libraries that I've seen tend to operate on the > streamlet kind of interface and b) It has a far smaller surface area(i.e. > number of interfaces) so might be easy to get something up quickly for > testing. > Thanks! > > On Mon, Oct 30, 2017 at 2:13 PM, Saikat Kanjilal <sxk1...@hotmail.com> > wrote: > > > Thanks Sanjeev, I had an initial idea that I wanted to float on the > list, > > I was thinking that as part of the initial scala port of the API I'd like > > to propose that we use this tool: http://javatoscala.com/ Java to Scala converter<http://javatoscala.com/> javatoscala.com How does it work? Java to Scala converter is created with Play framework and Scalagen library. I don't want you to see my code. No problem. Java to Scala converter ... > Java to Scala converter<http://javatoscala.com/> Java to Scala converter<http://javatoscala.com/> javatoscala.com How does it work? Java to Scala converter is created with Play framework and Scalagen library. I don't want you to see my code. No problem. Java to Scala converter ... > javatoscala.com > How does it work? Java to Scala converter is created with Play framework > and Scalagen library. I don't want you to see my code. No problem. Java to > Scala converter ... > > > > > > > Java to Scala converter<http://javatoscala.com/> Java to Scala converter<http://javatoscala.com/> javatoscala.com How does it work? Java to Scala converter is created with Play framework and Scalagen library. I don't want you to see my code. No problem. Java to Scala converter ... > Java to Scala converter<http://javatoscala.com/> Java to Scala converter<http://javatoscala.com/> javatoscala.com How does it work? Java to Scala converter is created with Play framework and Scalagen library. I don't want you to see my code. No problem. Java to Scala converter ... > javatoscala.com > How does it work? Java to Scala converter is created with Play framework > and Scalagen library. I don't want you to see my code. No problem. Java to > Scala converter ... > > > > > javatoscala.com > > How does it work? Java to Scala converter is created with Play framework > > and Scalagen library. I don't want you to see my code. No problem. Java > to > > Scala converter ... > > > > > > I propose that we use the above tool to do an initial conversion of all > > the Java functional and oo interfaces and then plug that into the initial > > design doc as well as the codebase to get things off the ground. I have > > already forked the heron repo and will be doing this off of a child > branch > > based on my fork. > > > > > > How does that sound to folks, anyone has strong objections, of course I > > fully realize that I will still need to do a lot of work around > refactoring > > the code even when using the converter so I'm prepared for that, the goal > > in using the converters is to save a bit of time so that manual > > intervention isnt necessarily needed in designing every interface. > > > > Thoughts. > > > > ________________________________ > > From: Sanjeev Kulkarni <sanjee...@gmail.com> > > Sent: Sunday, October 29, 2017 10:18 PM > > To: dev@heron.incubator.apache.org > > Subject: Re: Hello heron community > > > > Hi Saikat, > > Welcome to the Heron users group. Your plan of action sounds good to me. > A > > native scala interface would be great for all those scala fans. Eagerly > > awaiting design doc. > > Thanks! > > > > On Sun, Oct 29, 2017 at 6:17 PM, Saikat Kanjilal <sxk1...@hotmail.com> > > wrote: > > > > > Hello Folks, > > > > > > I'm interested in taking and driving the following github component > (this > > > means design/development/architecture and more). > > > > > > > > > https://github.com/twitter/heron/issues/668 [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://github.com/twitter/heron/issues/668> Ability to write native scala topologies · Issue #668 · twitter/heron<https://github.com/twitter/heron/issues/668> github.com Support for Scala Instance and APIs. > > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https:// > > github.com/twitter/heron/issues/668> > > > > Ability to write native scala topologies · Issue #668 · twitter/heron< > > https://github.com/twitter/heron/issues/668> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://github.com/twitter/heron/issues/668> Ability to write native scala topologies · Issue #668 · twitter/heron<https://github.com/twitter/heron/issues/668> github.com Support for Scala Instance and APIs. > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https:// > github.com/twitter/heron/issues/668> > > Ability to write native scala topologies · Issue #668 · twitter/heron< > https://github.com/twitter/heron/issues/668> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://github.com/twitter/heron/issues/668> Ability to write native scala topologies · Issue #668 · twitter/heron<https://github.com/twitter/heron/issues/668> github.com Support for Scala Instance and APIs. > github.com > Support for Scala Instance and APIs. > > > > > github.com > > Support for Scala Instance and APIs. > > > > > > > > > > > > > > > [https://avatars1.githubusercontent.com/u/367684?v=4&s=400]<https:// > > > github.com/twitter/heron/issues/668> > > > > > > Ability to write native scala topologies · Issue #668 ...< > > > https://github.com/twitter/heron/issues/668> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://github.com/twitter/heron/issues/668> Ability to write native scala topologies · Issue #668 · twitter/heron<https://github.com/twitter/heron/issues/668> github.com Support for Scala Instance and APIs. > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https:// > github.com/twitter/heron/issues/668> > > Ability to write native scala topologies · Issue #668 · twitter/heron< > https://github.com/twitter/heron/issues/668> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://github.com/twitter/heron/issues/668> Ability to write native scala topologies · Issue #668 · twitter/heron<https://github.com/twitter/heron/issues/668> github.com Support for Scala Instance and APIs. > github.com > Support for Scala Instance and APIs. > > > > > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https:// > > github.com/twitter/heron/issues/668> > > > > Ability to write native scala topologies · Issue #668 · twitter/heron< > > https://github.com/twitter/heron/issues/668> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://github.com/twitter/heron/issues/668> Ability to write native scala topologies · Issue #668 · twitter/heron<https://github.com/twitter/heron/issues/668> github.com Support for Scala Instance and APIs. > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https:// > github.com/twitter/heron/issues/668> > > Ability to write native scala topologies · Issue #668 · twitter/heron< > https://github.com/twitter/heron/issues/668> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://github.com/twitter/heron/issues/668> Ability to write native scala topologies · Issue #668 · twitter/heron<https://github.com/twitter/heron/issues/668> github.com Support for Scala Instance and APIs. > github.com > Support for Scala Instance and APIs. > > > > > github.com > > Support for Scala Instance and APIs. > > > > > > > > > github.com > > > heron - Heron is a realtime, distributed, fault-tolerant stream > > processing > > > engine from Twitter > > > > > > > > > > > > > > > I've joined the heron slack channel as recommended and am ready to > start > > > by creating a master JIRA issue to track this if not already done, I > > wanted > > > some guidance from a committer or two to help guide me a bit take care > of > > > at he administrative details and how to get this kicked off the ground. > > My > > > background includes working on apache reef codebase as well as dabbling > > in > > > apache mahout and apache flume as well as storm and various nosql/graph > > > databases. Looking forward to being part of the community. > > > > > > > > > > > > The rough plan of action in my mind so you know: > > > > > > 1) Review heron architecture and codebase in detail > > > > > > 2) Create an umbrella JIRA with the first series of tasks > > > > > > 3) Come up with a design doc > > > > > > 4) Build the interfaces > > > > > > 5) Add more dev tasks to JIRA > > > > > > 6) Drive the implementation > > > > > > > > > Thanks in advance for your help. > > > > > > > > >
