MySpace recently released their map-reduce implementation as opensource (it's .NET based). MySpace as you might know is one of the few big websites that runs on Windows.
http://code.google.com/p/qizmt/ On Thu, Sep 17, 2009 at 10:42 PM, Steve Loughran <ste...@apache.org> wrote: > Bill Habermaas wrote: > >> It's interesting that Hadoop, being written entirely in Java, has such a >> spotty reputation running on different platforms. I had to patch it to run >> on AIX and need cygwin (gack!) so it will run on Windows. I'm surprised >> nobody has thought about removing it's use of bash to run system commands >> (which is NOT especially portable). Now that Hadoop only comes only in a >> Java 1.6 flavor why can't it figure out disk space using the native java >> runtime instead of executing the DF command under bash? Of course it runs >> other system commands as well which in my opinion isn't too cool. >> > > > It is run at scale on big linux systems, and they are the ones that > encounter problems with 16GB heaps and exec(), various other JVM quirks that > lead the developers to say Linux + Sun JVM only. You are free to use other > operating systems and even JVMs (I've used JRockit with some minor logging > problems in test runs), but you get to encounter the problems. You can and > should submit patches back, but if you diverge from the approved standard, > you get to retest at scale, because nobody else is going to do it for you. > > Supporting different unix versions is much easier than supporting > windows+linux/unix, especially if you are trying to do high availability > stuff, integrate with management tools, etc. I think it would be nice if > Hadoop would build and run standalone on Windows without cygwin, but for all > other actions, a more ruthless "Unix-ish only" would be harsh but make it > easier to manage problems. > > Even in a Linux-only world, you are left with the "which distro", question > -were there to be official apache Hadoop RPMs and .deb files, there'd be > discussions about which platforms to support. RHEL+Centos 5.X would be the > obvious choice, but what else? > > -steve > -- Harish Mallipeddi http://blog.poundbang.in
