Because a lot of people here are using HDFS day in and day out the following might be quite interesting for some.
Magna Tempus Group has just rolled out a readily available Spark 0.5 (www.spark-project.org) packaged for Ubuntu distribution. Spark delivers up to 20x faster experience (sic!) using in-memory analytics and a computational model that is different from MapReduce. Our package is build against current official Apache Hadoop 1.0.3, so it should be compatible with everything from 0.20.205 up to Hadoop 1.1 release candidate. Redhat/CentOS version is coming in a few days (in case someone is interested). You can find all related information at http://www.magnatempusgroup.net/blog/2012/09/24/incredibly-fast-in-memory-analytics-for-bigdata-technology-preview/ and download installable package from http://magnatempusgroup.net/ftphost/releases/Spark-0.5-1.0.3/ Compatible Hadoop 1.0.3 stack is available for direct installation from our repository http://www.magnatempusgroup.net/blog/2012/05/25/mtg-release-0-3-1/ The package is created in the exact standards of BigTop stack, so it is quite standard distribution package. We would love to hear your feedback and comments! -- With regards, Alef MTG development team