Hey Josh, Nice to see some interest, I just pushed from my local repo with several bigger changes. I've separated crackle into 3 parts core, hbase and example on my todo list: - jar file assembly, currently I'm using jar command from shell to create the job jar, this obviously needs to be rewritten in order to make crackle portable - I need to add support for all sources and targets you have in crunch - need to integrate crunch hbase: sources, targets and types
after these are done, some nice to do tasks: - cannot define mr pipelines from clojure REPL, although crackle compiles pipeline classes on the fly it still needs the code to be written to a local file, so it's not as nice as it should be - DSL sucks: * in current shape you don't have access to PObjects from intermediate steps * users have to know crunch api very well otherwise they will get confused: what type goes where and why they have to use this particular function type Regards PS I'm also a clojure noob, I did learn common lisp several years ago but playing with clojure only for several months On Mon, Nov 26, 2012 at 11:48 PM, Josh Wills <[email protected]> wrote: > Victor, > > Just got my own personal fork-- congrats on getting the MR pipeline impl > working. What needs doing? Keep in mind that I'm a total clojure n00b, > despite repeated encouragement from lots of developers I respect and > admire. > > Josh > > > On Tue, Nov 20, 2012 at 2:33 PM, Victor Iacoban <[email protected] > >wrote: > > > Hi, > > > > I have the basics done here: > > https://github.com/viacoban/crackle > > > > It's only MemPipeline for now, still have to build the jar in background > > for MRPipeline, but before going there I have a small issue to solve. > > > > So if anyone has written several clojure macroses or know somebody who > did > > please write to me directly and we will take it from there > > > > Any comments or input is welcome > > > > Victor > > > > > > -- > Director of Data Science > Cloudera <http://www.cloudera.com> > Twitter: @josh_wills <http://twitter.com/josh_wills> >
