Andrew et al,I wanted to ask about a few items while I'm researching my dev proposal, so what I'm looking to build is a streaming analytics platform to do things like collaborative filtering and anomaly detection on large amounts of streaming data that are either generated from events (kafka) or through a firehose like Amazon Kinesis, my initial thinking is that this pipe of events/data would be connected to a rest API that sits on top of mahout, the backend underneath mahout would use a hybrid form of spark as well as spark streaming, I'm wondering whether Samsara was designed from the ground up to deal with large amounts of streaming data or whether this is not a use case targeted yet. My goal is to build a platform with several data sources/sinks and produce intermediate checkpoints where transformations are applied to the data before once again sending to a set of sinks/sources. Therefore the potential fits into and out of mahout include: 1) A rest API that leverages spray and akka and invokes one or more algorithms in mahout2) A runtime environment with scala actors that allows one to either ingest data or perform transformations on data through the use of various classification and clustering algorithms, the runtime environment would ingest algorithms using mahout as a library3) A rich set of actors dealing with various no sql and graph based datastores (cassandra/neo4j/titan/mongo)
Some insight into Samsara would be great as I'm trying to understand the entry points into mahout. Thanks in advance. > From: ap....@outlook.com > To: dev@mahout.apache.org > Subject: Re: Mahout contributions > Date: Thu, 28 Apr 2016 21:43:19 +0000 > > I don't think that this sort of of integration work would be a good fit > directly to the Mahout project. Mahout is more about math, algorithms and an > environment to develop algorithms. We stay away from direct platform > integration. In the past we did have some elasticsearch/mahout integration > work that is not in the code base for this exact reason. I would suggest > that better places to contribute something like this may be: PIO > (https://prediction.io/), or even directly as a package for spark > http://spark-packages.org/ . > > Recent projects integrating Mahout have recently been added to PIO: > https://github.com/PredictionIO/template-scala-parallel-universal-recommendation. > > > I think that the project that you are proposing would be a better fit there. > > Thanks, > > Andy > > > ________________________________________ > From: Saikat Kanjilal <sxk1...@hotmail.com> > Sent: Thursday, April 28, 2016 1:50 PM > To: dev@mahout.apache.org > Subject: Re: Mahout contributions > > I want to start with social data as an example, for example data returned > from FB graph API as well user Twitter data, will send some samples later if > you're interested. > > Sent from my iPhone > > > On Apr 28, 2016, at 10:41 AM, Khurrum Nasim <khurrum.na...@useitc.com> > > wrote: > > > > > > What type of JSON payload size are we talking about here ? > > > >> On Apr 28, 2016, at 1:32 PM, Saikat Kanjilal <sxk1...@hotmail.com> wrote: > >> > >> Because EL gives you the visualization and non Lucene type query > >> constructs as well and also that it already has a rest API that I plan on > >> tying into mahout. I plan on wrapping some of the clustering algorithms > >> that I implement using Mahout and Spark as a service which can then make > >> calls into other services (namely elasticsearch and neo4j graph service). > >> > >> Sent from my iPhone > >> > >>> On Apr 28, 2016, at 10:22 AM, Khurrum Nasim <khurrum.na...@useitc.com> > >>> wrote: > >>> > >>> @Saikat- why use EL instead of Lucene directly. > >>> > >>> > >>> > >>>> On Apr 28, 2016, at 12:08 PM, Saikat Kanjilal <sxk1...@hotmail.com> > >>>> wrote: > >>>> > >>>> This is great information thank you, based on this recommendation I > >>>> won't create a JIRA but start work on my project and when the code > >>>> approaches the percentages you are describing I will create the > >>>> appropriate JIRA's and put together a proposal to send to the list, > >>>> sound ok? Based on your latest updates to the wiki i will work on a > >>>> handful of the clustering algorithms since I see that the Spark > >>>> implementations for these are not yet complete. > >>>> Thank you again > >>>> > >>>>> From: ap....@outlook.com > >>>>> To: dev@mahout.apache.org > >>>>> Subject: Re: Mahout contributions > >>>>> Date: Thu, 28 Apr 2016 01:31:09 +0000 > >>>>> > >>>>> Saikat, > >>>>> > >>>>> One other thing that I should say is that you do not need clearance or > >>>>> input from the committers to begin work on your project, and the > >>>>> interest can and should come from the community as a whole. You can > >>>>> write proposal as you've done, and if you don't see any "+1"s or > >>>>> responses from the community at whole with in a few days, you may want > >>>>> to explain in more detail, give examples and use cases. If you are > >>>>> still not seeing +1s or any responses from others then I think you can > >>>>> assume that there may not be interest; this is usually how things work. > >>>>> > >>>>> However if its something that your passionate about and you feel like > >>>>> you can deliver this should not to stop you. People do not always read > >>>>> the dev@ emails or have time to respond. You can still move forward > >>>>> with your proposed contribution by following the steps laid out in my > >>>>> previous email; follow the protocol at: > >>>>> > >>>>> http://mahout.apache.org/developers/how-to-contribute.html > >>>>> > >>>>> and create a JIRA. When you have reached a significant amount of > >>>>> completion (around 70-80%), open a PR for review, this way you can > >>>>> explain in more detail. > >>>>> > >>>>> But please realize that when you open a JIRA for a new issue there is > >>>>> some expectation of a commitment on your part to complete it. > >>>>> > >>>>> For example, I am currently investigating some new plotting features. > >>>>> I have spent a good deal of time this week and last already and am even > >>>>> mocking up code as a sketch of what may become an implementation before > >>>>> I open a "New Feature" JIRA for it. > >>>>> > >>>>> My point is absolutely not to discourage you or anybody else from > >>>>> opening JIRAs for new features, rather to let you know that when you > >>>>> open an JIRA for a new issue, It tells others that your are working on > >>>>> it, and thus may discourage another with a similar idea to contribute > >>>>> this feature. So it is best to open it once you've begun your work and > >>>>> are committed to it. > >>>>> > >>>>> Andy > >>>>> > >>>>> ________________________________________ > >>>>> From: Saikat Kanjilal <sxk1...@hotmail.com> > >>>>> Sent: Wednesday, April 27, 2016 8:24 PM > >>>>> To: dev@mahout.apache.org > >>>>> Subject: RE: Mahout contributions > >>>>> > >>>>> Andrew,Thank you very much for your input, I actually want to start a > >>>>> new set of JIRAs, here's what I want to work on, I want to build a > >>>>> framework that ties together search/visualization capability with some > >>>>> machine learning algorithms, so essentially think of it as tying in > >>>>> elasticsearch and kibana into mahout , the user can search for their > >>>>> data with elasticsearch and for deeper analysis on that data they can > >>>>> feed that data into one or more mahout backends for analysis. Another > >>>>> interesting tie in might be to hack kibana to render ggplot like > >>>>> graphics based on the output of mahout algorithms (assuming this can be > >>>>> a kibana plugin). > >>>>> Before I go hog wild to create a bunch of JIRA's I'd like to know if > >>>>> there's interest in this initiative. The tool will bring together the > >>>>> ELK stack with dynamic machine learning algorithms. I can go into a > >>>>> lot more detail around use cases if there's enough interest. > >>>>> Looking forward to your and other committers input.Thanks > >>>>> > >>>>>> From: ap....@outlook.com > >>>>>> To: dev@mahout.apache.org > >>>>>> Subject: Re: Mahout contributions > >>>>>> Date: Wed, 27 Apr 2016 20:16:38 +0000 > >>>>>> > >>>>>> Hello Saikat, > >>>>>> > >>>>>> #1 and #2 above are already implemented. #4 is tricky so i would not > >>>>>> recommend without a strong knowledge of the codebase, and #5 is now > >>>>>> deprecated. (I've just updated the algorithms grid to reflect this). > >>>>>> The algorithms page includes both algorithms implemented in the > >>>>>> math-scala library and algorithms which have CLI drivers written for > >>>>>> them. > >>>>>> > >>>>>> Please see: http://mahout.apache.org/developers/how-to-contribute.html > >>>>>> > >>>>>> And please note that per that documentation, it is in everybody's best > >>>>>> interest to keep messages on list, contacting committers directly is > >>>>>> discouraged. > >>>>>> > >>>>>> The best way to contribute (if you have not found a new bug or issue) > >>>>>> would be for you to pick a single open issue in the mahout JIRA which > >>>>>> is not already assigned, and start work on it. When your work is > >>>>>> ready for review, just open up a PR and the committers will review it. > >>>>>> Please note that if you do pick up an issue to work on, we do expect > >>>>>> some amount of responsibility and reliability and tangible amount of > >>>>>> satisfactory work since once you've marked a JIRA as something you're > >>>>>> working on, others will pass on it. > >>>>>> > >>>>>> Another good way to contribute would be to look for enhancements that > >>>>>> could make to existing code not necessarily open JIRAs that need to be > >>>>>> assigned to you. For example please see the recent contribution and > >>>>>> workflow on: https://issues.apache.org/jira/browse/MAHOUT-1833 . > >>>>>> > >>>>>> If you have something new that you'd like to implement, simply start a > >>>>>> new JIRA issue and begin work on it. In this case, when you have some > >>>>>> code that is ready for review, you can simply open up a PR for it and > >>>>>> committers will review it. For new implementations, we generally say > >>>>>> that you should do this when you are at least 70-80% finished with > >>>>>> your coding. > >>>>>> > >>>>>> Thank You, > >>>>>> > >>>>>> Andy > >>>>>> > >>>>>> > >>>>>> > >>>>>> ________________________________________ > >>>>>> From: Saikat Kanjilal <sxk1...@hotmail.com> > >>>>>> Sent: Tuesday, April 26, 2016 7:17 PM > >>>>>> To: dev@mahout.apache.org > >>>>>> Subject: RE: Mahout contributions > >>>>>> > >>>>>> Hello,Following up on my last email with more specifics, I've looked > >>>>>> through the wiki > >>>>>> (https://mahout.apache.org/users/basics/algorithms.html) and I'm > >>>>>> interested in implementing the one or more of the following algorithms > >>>>>> with Mahout using spark: 1) Matrix Factorization with ALS 2) Naive > >>>>>> Bayes 3) Weighted Matrix Factorization, SVD++ 4) Sparse TF-IDF Vectors > >>>>>> from Text 5) Lucene integration. > >>>>>> Had a few questions:1) Which of these should I start with and where is > >>>>>> there the greatest need?2) Should I fork the repo and create branches > >>>>>> for the each of the above implementations?3) Should I go ahead and > >>>>>> create some JIRAs for these? > >>>>>> Would love to have some pointers to get started?Regards > >>>>>> > >>>>>> From: sxk1...@hotmail.com > >>>>>> To: dev@mahout.apache.org > >>>>>> Subject: Mahout contributions > >>>>>> Date: Wed, 30 Mar 2016 10:23:45 -0700 > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> Hello Committers,I was looking through the current jira tickets and > >>>>>> was wondering if there's a particular area of Mahout that needs some > >>>>>> more help than others, should I focus on contributing some algorithms > >>>>>> usign DSL or Samsara related efforts, I've finally got some bandwidth > >>>>>> to do some work and would love some guidance before assigning myself > >>>>>> some tickets.Regards > >
