Did you try Pig ? It drastically reduces boiler plate code for common operations like, Join, Group, Cogroup, Filter, Projection, Order Pig also gives you some advanced stuff like Multi-query optimization that is ugly to code my hand and difficult to maintain.
Most of the people that I know, don't write Map-reduce code anymore as its too low level and simply too much work. If you are looking for alternative stuff to solve your problems better, I suggest you give Pig (version 0.8) a shot - http://pig.apache.org Regards -@nkur On 4/7/11 9:48 AM, "Guy Doulberg" <guy.doulb...@conduit.com> wrote: Thanks for your answers, I checked cascading for a while, It was easy to get started and to do the tutorial, I really liked the modeling of pipes, cogroups and so on... But when I tried to implement a real life project, things has become too complicated to me, things didn't go the way I expected them to go, I had to implement it using plain map/red....api I think I should give it a try again. -----Original Message----- From: Guy Doulberg [mailto:guy.doulb...@conduit.com] Sent: Thursday, April 07, 2011 10:40 AM To: common-user@hadoop.apache.org Subject: Developing, Testing, Distributing Hey, I have been developing Map/Red jars for a while now, and I am still not comfortable with the developing environment I gathered for myself (and the team) I am curious how other Hadoop developers out-there, are developing their jobs... What IDE you are using, What plugins to the IDE you are using How do you test your code, which Unit test libraries your using, how do you run your automatic tests after you have finished the development? Do you have test/qa/staging environments beside the dev and the production? How do you keep it similar to the production Code reuse - how do you build components that can be used in other jobs, do you build generic map or reduce class? I can tell you that I have no answer to the questions above, I hope that this post is not too general, but I think the discussion here could be helpful for newbie and experienced developers all together Thanks Guy
