Also interested in this project, but low on time for a few weeks. One quick bit of feedback: I strongly suspect that there is a way to eliminate all the type related code from Clojure, probably by using Macros
-- Joe On Wed, Nov 28, 2012 at 9:07 AM, Matthias Friedrich <[email protected]> wrote: > Hi Victor, > > just for the record, I'm also very interested in this. It's just that > in the time before christmas, things are really busy at work. But > I'll definitely play around with crackle. > > Thanks, > Matthias > > On Tuesday, 2012-11-27, Victor Iacoban wrote: > > Hey Josh, > > > > Nice to see some interest, I just pushed from my local repo with several > > bigger changes. I've separated crackle into 3 parts core, hbase and > example > > on my todo list: > > - jar file assembly, currently I'm using jar command from shell to create > > the job jar, this obviously needs to be rewritten in order to make > crackle > > portable > > - I need to add support for all sources and targets you have in crunch > > - need to integrate crunch hbase: sources, targets and types > > > > after these are done, some nice to do tasks: > > - cannot define mr pipelines from clojure REPL, although crackle compiles > > pipeline classes on the fly it still needs the code to be written to a > > local file, so it's not as nice as it should be > > - DSL sucks: > > * in current shape you don't have access to PObjects from intermediate > > steps > > * users have to know crunch api very well otherwise they will get > > confused: what type goes where and why they have to use this particular > > function type > > > > Regards > > > > PS I'm also a clojure noob, I did learn common lisp several years ago but > > playing with clojure only for several months > > > > > > On Mon, Nov 26, 2012 at 11:48 PM, Josh Wills <[email protected]> > wrote: > > > > > Victor, > > > > > > Just got my own personal fork-- congrats on getting the MR pipeline > impl > > > working. What needs doing? Keep in mind that I'm a total clojure n00b, > > > despite repeated encouragement from lots of developers I respect and > > > admire. > > > > > > Josh > > > > > > > > > On Tue, Nov 20, 2012 at 2:33 PM, Victor Iacoban < > [email protected] > > > >wrote: > > > > > > > Hi, > > > > > > > > I have the basics done here: > > > > https://github.com/viacoban/crackle > > > > > > > > It's only MemPipeline for now, still have to build the jar in > background > > > > for MRPipeline, but before going there I have a small issue to solve. > > > > > > > > So if anyone has written several clojure macroses or know somebody > who > > > did > > > > please write to me directly and we will take it from there > > > > > > > > Any comments or input is welcome > > > > > > > > Victor > > > > > > > > > > > > > > > > -- > > > Director of Data Science > > > Cloudera <http://www.cloudera.com> > > > Twitter: @josh_wills <http://twitter.com/josh_wills> > > > >
