On Sun, Nov 15, 2009 at 6:56 PM, Mark Kerzner <markkerz...@gmail.com> wrote:
> Oops, I just switched to 0.19. There is no .20 on the EC2 that I see. Can I > risk it - I have few MR-specific thing - or go back to 0.18? > > You can use 0.20 on EC2 using Cloudera's scripts. I wouldn't necessarily recommend moving back to 0.18.3 at this point for new development. For new projects, unless you have need for the absolute most stable release, I'd recommend 0.20.1, which is being used successfully in production by many organizations. -Todd > On Sun, Nov 15, 2009 at 8:47 PM, Todd Lipcon <t...@cloudera.com> wrote: > > > Hi Mark, > > > > You should be good to go. > > > > However, I wouldn't recommend running 0.19 at all. Upgrade to 0.20 - 19 > has > > lots of bugs and I know of few people running it in production - 0.19.0 > > especially. > > > > -Todd > > > > On Sun, Nov 15, 2009 at 4:30 PM, Mark Kerzner <markkerz...@gmail.com> > > wrote: > > > > > A bit more specifically, I am planning to use 0.19.0 for EC2 cluster, > > > because that's the only .19 version available on EC2, and the version > on > > my > > > workstation is 0.19.2, because it was easily available for download. > > Since > > > my complete cluster runs on 0.19.0, and my jar is also there, I should > > have > > > no problem, right? > > > > > > Thank you, > > > Mark > > > > > > On Sun, Nov 15, 2009 at 3:54 PM, Todd Lipcon <t...@cloudera.com> > wrote: > > > > > > > Hi Mark, > > > > > > > > The simple answer is "yes, to be safest, they should match". > > > > > > > > In truth, the answer is a bit more complex. Since Java is dynamically > > > > "linked" (classloaded) at runtime, as long as the method signatures > and > > > > class names you're using in your code haven't changed between > versions, > > > > your > > > > jar compiled against one version will run against another. Between > > > > different > > > > versions of the same major release (eg 0.20 to 0.20.1) this is almost > > > > always > > > > the case except for the occasional backported new API in a later > > version. > > > > So, if you've compiled against 0.20.0 and then run against 0.20.1, or > a > > > > Cloudera build like 0.20.1+152 you should be fine. > > > > > > > > If you're using a jar compiled against 0.18.3 and trying to run on > > 0.20, > > > > though, your luck will be much more varied. Most of the APIs from > > 0.18.3 > > > > are > > > > still present but there are a few things that will break with strange > > > > errors. So, I *strongly* recommend that you compile against the same > > > major > > > > version you plan on running against. > > > > > > > > All of the above is referring to the case when you're compiling a jar > > > > locally, and then copying it to the cluster and using the cluster's > > > "hadoop > > > > jar foo.jar" command to submit the job. If you're trying to use your > > > local > > > > Hadoop installation pointed at the jobtracker from the remote > cluster, > > > the > > > > requirements are a bit more strict - the protocol version numbers > must > > > > match, which means you *must* run the same major release (0.18 hadoop > > > > cannot > > > > submit to an 0.20 cluster). > > > > > > > > Thanks > > > > -Todd > > > > > > > > On Sun, Nov 15, 2009 at 1:37 PM, Mark Kerzner <markkerz...@gmail.com > > > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > when I am building my jar for a MapReduce job, I include the > version > > of > > > > > Hadoop I am running on my workstation. When I run Hadoop on a > > cluster, > > > > > there > > > > > is a version that runs on a cluster. Do they have to match? In > other > > > > words, > > > > > how does my Hadoop jar interacts with the cluster's Hadoop? > > > > > > > > > > Thank you, > > > > > Mark > > > > > > > > > > > > > > >
