[ASK] How to unify methodological prototyping with implementation in reality (aka in cascalog)?
Hi everybody, I have been doing some random machine learning doodling and have always been haunted with this problem. When I just wanna set out and try if my methodology would work, I would program in Clojure aiming at a small dataset available within one single host memory (or even just doodle R or Python. I know what's blasphemy). When I wanna implement the results, I would usually have to re-write the whole process in Cascalog just in order to apply them in Hadoop data. I know I can still just start up to write Hadoop functions. But that's a bit overshoot since I have to run simulation study to verify the performance, correctness etc before even going into production development. Besides, having hadoop in memory to sit between my program and the computer just slows down computation. I've day-dreamed a lot some smart ways to do prototyping and, with the flip of a switch, my functions just become easily applicable to Hadoop data. Wondering if there's any ways to do that? I actually asked around and some one who I admire a lot suggested writing a Hadoop emulator-like wrapper for my input data. Wondering if there's an easier way? I would appreciate any input. Thanks. Hesen -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [ASK] How to unify methodological prototyping with implementation in reality (aka in cascalog)?
If I'm not mistaken, PigPen allows you to execute queries locally, without a hadoop cluster etc. Does that solve your problem? https://github.com/Netflix/PigPen/wiki/Design-and-Features#testing-local-execution-and-debugging On Mon, May 26, 2014 at 3:26 AM, Hesen Peng wrote: > Hi everybody, > > I have been doing some random machine learning doodling and have always > been haunted with this problem. > > When I just wanna set out and try if my methodology would work, I would > program in Clojure aiming at a small dataset available within one single > host memory (or even just doodle R or Python. I know what's blasphemy). > > When I wanna implement the results, I would usually have to re-write the > whole process in Cascalog just in order to apply them in Hadoop data. I > know I can still just start up to write Hadoop functions. But that's a bit > overshoot since I have to run simulation study to verify the performance, > correctness etc before even going into production development. Besides, > having hadoop in memory to sit between my program and the computer just > slows down computation. > > I've day-dreamed a lot some smart ways to do prototyping and, with the > flip of a switch, my functions just become easily applicable to Hadoop > data. Wondering if there's any ways to do that? > > I actually asked around and some one who I admire a lot suggested writing > a Hadoop emulator-like wrapper for my input data. Wondering if there's an > easier way? I would appreciate any input. Thanks. > > Hesen > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- Kind Regards, Atamert Ölçgen -+- --+ +++ www.muhuk.com -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [ASK] How to unify methodological prototyping with implementation in reality (aka in cascalog)?
That's exactly the tool. Thanks a lot :) On Sun, May 25, 2014 at 10:07 PM, Atamert Ölçgen wrote: > If I'm not mistaken, PigPen allows you to execute queries locally, without > a hadoop cluster etc. Does that solve your problem? > > > https://github.com/Netflix/PigPen/wiki/Design-and-Features#testing-local-execution-and-debugging > > > > On Mon, May 26, 2014 at 3:26 AM, Hesen Peng wrote: > >> Hi everybody, >> >> I have been doing some random machine learning doodling and have always >> been haunted with this problem. >> >> When I just wanna set out and try if my methodology would work, I would >> program in Clojure aiming at a small dataset available within one single >> host memory (or even just doodle R or Python. I know what's blasphemy). >> >> When I wanna implement the results, I would usually have to re-write the >> whole process in Cascalog just in order to apply them in Hadoop data. I >> know I can still just start up to write Hadoop functions. But that's a bit >> overshoot since I have to run simulation study to verify the performance, >> correctness etc before even going into production development. Besides, >> having hadoop in memory to sit between my program and the computer just >> slows down computation. >> >> I've day-dreamed a lot some smart ways to do prototyping and, with the >> flip of a switch, my functions just become easily applicable to Hadoop >> data. Wondering if there's any ways to do that? >> >> I actually asked around and some one who I admire a lot suggested writing >> a Hadoop emulator-like wrapper for my input data. Wondering if there's an >> easier way? I would appreciate any input. Thanks. >> >> Hesen >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clojure@googlegroups.com >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+unsubscr...@googlegroups.com. >> >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Kind Regards, > Atamert Ölçgen > > -+- > --+ > +++ > > www.muhuk.com > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to a topic in the > Google Groups "Clojure" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/clojure/pw-xPg4D-DA/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- Hesen Peng 彭河森 -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
