Hi Thomas, On Sun, Mar 12, 2017 at 10:13 AM, Thomas Weise <[email protected]> wrote:
> I think a flat list with good naming would make it easier to find a > matching example. > I was looking at how examples are laid out in other projects like spark and saw that there was top level classification such as r, ml, sql, streaming etc., so I thought a top level folder, where there are lot of common themed examples, such as datat movement, would be a good way to reduce clutter. > > topnwords ? There are already two related examples in Malhar: wordcount and > twitter > It need not be brought over. Can some of the examples be consolidated into a single example package, > fileioXXX/fileoutput look like candidates. > I will look into it. Thanks Lakshmi Prasanna > > > On Sat, Mar 11, 2017 at 6:36 PM, Lakshmi Velineni <[email protected] > > > wrote: > > > Hi, > > > > I have come up with the following list of applications to move from > > datatorrent/examples into apex-malhar/examples. There were many examples > > that do a similar functionality of moving data around from different > > sources to destinations, so I separated them all into a separate folder > > called "Ingestion". > > > > csvformatter > > > > dedup > > > > dynamic-partition > > > > enricher > > > > exactly-once > > > > filter > > > > innerjoin > > > > maprapp > > > > parser > > > > partition > > > > recordreader > > > > throttle > > > > topnwords > > > > transform > > > > unifiers > > > > *Ingestion/* > > > > cassandrainput > > > > cassandraoutput > > > > fileio > > > > fileio-multidir > > > > fileio-simple > > > > fileoutput > > > > filetojdbc > > > > hdfs2kafka > > > > hdfs-sync > > > > jdbcingest > > > > jdbctijdbc > > > > jmsActiveMQ > > > > jmsSqs > > > > kafka > > > > s3-to-hdfs-sync > > > > s3output > > > > > > Let me know your thoughts. > > > > > > Thanks > > > > Lakshmi Prasanna > > > > > > > > > > > > > > > > On Thu, Feb 23, 2017 at 10:34 AM, Amol Kekre <[email protected]> > wrote: > > > > > Yes, should merge samples into examples. Ideally the names of the > > examples > > > should be more descriptive in terms of "how to" as opposed to a title. > PI > > > demo for example has lots of ways to do compute. So it could be named > as > > > "pi - distributed compute" in examples. Similarly if others can be > named > > to > > > bring out features to look at from the examples, it would make examples > > > more useful to readers. > > > > > > Thks > > > Amol > > > > > > > > > > > > E:[email protected] | M: 510-449-2606 | Twitter: @*amolhkekre* > > > > > > www.datatorrent.com | apex.apache.org > > > > > > *Join us at Apex Big Data World-San Jose > > > <http://www.apexbigdata.com/san-jose.html>, April 4, 2017!* > > > [image: http://www.apexbigdata.com/san-jose-register.html] > > > <http://www.apexbigdata.com/san-jose-register.html> > > > > > > On Thu, Feb 23, 2017 at 10:07 AM, Sanjay Pujare < > [email protected]> > > > wrote: > > > > > > > + for renaming to examples. While we are at it, how about merging > > > "samples" > > > > also in the new "examples" ? > > > > > > > > On Thu, Feb 23, 2017 at 9:47 AM, Munagala Ramanath < > > [email protected]> > > > > wrote: > > > > > > > > > +1 for renaming to "examples" > > > > > > > > > > Ram > > > > > > > > > > On Thu, Feb 23, 2017 at 9:12 AM, Lakshmi Velineni < > > > > [email protected] > > > > > > > > > > > wrote: > > > > > > > > > > > I am ready to bring the examples over into the demos folder. I > was > > > > > > wondering if anybody has any input on Thomas's suggestion to > rename > > > the > > > > > > demos folder to examples. I would rather do that first and then > > bring > > > > the > > > > > > examples over instead of doing it the other way around as that > > would > > > > lead > > > > > > to refactoring the new examples again. > > > > > > > > > > > > Thanks > > > > > > > > > > > > On Wed, Jan 25, 2017 at 8:12 AM, Lakshmi Velineni < > > > > > [email protected] > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > Since the examples have little history I was planning to have > two > > > > > > > commits for every example, one for the code as the primary > author > > > of > > > > > > > the example and another containing pom.xml and other changes to > > > make > > > > > > > it work under malhar. > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > On Wed, Nov 2, 2016 at 9:49 PM, Lakshmi Velineni > > > > > > > <[email protected]> wrote: > > > > > > > > Thanks for the suggestions and I am working on the process to > > > > migrate > > > > > > the > > > > > > > > examples with the guidelines you mentioned. I will send out a > > > list > > > > of > > > > > > > > examples and the destination modules very soon. > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Oct 27, 2016 at 1:43 PM, Thomas Weise < > > > > > [email protected]> > > > > > > > > wrote: > > > > > > > >> > > > > > > > >> Maybe a good first step would be to identify which examples > to > > > > bring > > > > > > > over > > > > > > > >> and where appropriate how to structure them in Malhar (for > > > > example, > > > > > I > > > > > > > see > > > > > > > >> multiple hdfs related apps that could go into the same Maven > > > > > module). > > > > > > > >> > > > > > > > >> > > > > > > > >> On Tue, Oct 25, 2016 at 1:00 PM, Thomas Weise < > [email protected] > > > > > > > > wrote: > > > > > > > >> > > > > > > > >> > That would be great. There are a few things to consider > when > > > > > working > > > > > > > on > > > > > > > >> > it: > > > > > > > >> > > > > > > > > >> > * preserve attribtion > > > > > > > >> > * ensure there is a test that runs the application in the > CI > > > > > > > >> > * check that dependencies are compatible license > > > > > > > >> > * maybe extract common boilerplate code from pom.xml > > > > > > > >> > > > > > > > > >> > etc. > > > > > > > >> > > > > > > > > >> > Existing examples are under https://github.com/apache/ > > > > > > > >> > apex-malhar/tree/master/demos > > > > > > > >> > > > > > > > > >> > Perhaps we should rename it to "examples" > > > > > > > >> > > > > > > > > >> > I also propose that each app has a README and we add those > > for > > > > > > > existing > > > > > > > >> > apps as well. > > > > > > > >> > > > > > > > > >> > Thanks, > > > > > > > >> > Thomas > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > On Tue, Oct 25, 2016 at 12:49 PM, Lakshmi Velineni < > > > > > > > >> > [email protected]> wrote: > > > > > > > >> > > > > > > > > >> >> Can i work on this? > > > > > > > >> >> > > > > > > > >> >> Thanks > > > > > > > >> >> Lakshmi Prasanna > > > > > > > >> >> > > > > > > > >> >> On Mon, Sep 12, 2016 at 9:41 PM, Ashwin Chandra Putta < > > > > > > > >> >> [email protected]> wrote: > > > > > > > >> >> > > > > > > > >> >> > Here is the JIRA: > > > > > > > >> >> > https://issues.apache.org/jira/browse/APEXMALHAR-2233 > > > > > > > >> >> > > > > > > > > >> >> > On Tue, Sep 6, 2016 at 10:20 PM, Amol Kekre < > > > > > > [email protected]> > > > > > > > >> >> wrote: > > > > > > > >> >> > > > > > > > > >> >> > > Good idea to consolidate them into Malhar. We should > > > bring > > > > in > > > > > > as > > > > > > > >> >> > > many > > > > > > > >> >> > from > > > > > > > >> >> > > this gitHub as possible. > > > > > > > >> >> > > > > > > > > > >> >> > > Thks > > > > > > > >> >> > > Amol > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > > >> >> > > On Tue, Sep 6, 2016 at 6:02 PM, Thomas Weise > > > > > > > >> >> > > <[email protected]> > > > > > > > >> >> > > wrote: > > > > > > > >> >> > > > > > > > > > >> >> > > > I'm also for consolidating these different example > > > > > locations. > > > > > > > We > > > > > > > >> >> should > > > > > > > >> >> > > > also look if all of it is still relevant. > > > > > > > >> >> > > > > > > > > > > >> >> > > > The stuff from the DT repository needs to be > brought > > > into > > > > > > shape > > > > > > > >> >> > > > wrt > > > > > > > >> >> > > > licensing, checkstyle, CI support etc. > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > On Tue, Sep 6, 2016 at 4:34 PM, Pramod Immaneni < > > > > > > > >> >> > [email protected]> > > > > > > > >> >> > > > wrote: > > > > > > > >> >> > > > > > > > > > > >> >> > > > > Sounds like a good idea. How about merging demos > > with > > > > > apps > > > > > > as > > > > > > > >> >> well? > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > On Tue, Sep 6, 2016 at 4:30 PM, Ashwin Chandra > > Putta > > > < > > > > > > > >> >> > > > > [email protected]> wrote: > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > Hi All, > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > We have a lot of examples for apex malhar > > operators > > > > in > > > > > > the > > > > > > > >> >> > following > > > > > > > >> >> > > > > > repository which resides outside of malhar. > > > > > > > >> >> > > > > > https://github.com/DataTorrent/examples/tree/ > > > > > > > master/tutorials > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > Now that it has grown quite a bit, does it make > > > sense > > > > > to > > > > > > > >> >> > > > > > bring > > > > > > > >> >> some > > > > > > > >> >> > > of > > > > > > > >> >> > > > > the > > > > > > > >> >> > > > > > most common examples to malhar repository? > > Probably > > > > > under > > > > > > > >> >> > > > > > apps > > > > > > > >> >> > > > directory? > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > That way folks looking at malhar repository > will > > > have > > > > > > some > > > > > > > >> >> samples > > > > > > > >> >> > to > > > > > > > >> >> > > > > look > > > > > > > >> >> > > > > > at without having to search elsewhere. > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > -- > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > Regards, > > > > > > > >> >> > > > > > Ashwin. > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > -- > > > > > > > >> >> > > > > > > > > >> >> > Regards, > > > > > > > >> >> > Ashwin. > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > _______________________________________________________ > > > > > > > > > > Munagala V. Ramanath > > > > > > > > > > Software Engineer > > > > > > > > > > E: [email protected] | M: (408) 331-5034 | Twitter: @UnknownRam > > > > > > > > > > www.datatorrent.com | apex.apache.org > > > > > > > > > > > > > > >
