Just to clarify because there seems to be some confusion: we are supporting any Spark version >=1.4 and Hadoop version >=2.4. The only reason why our build sticks to older versions is to avoid exploiting new features too eagerly which would loose backwards compatibility if done without care.
With regard to Hadoop v1: Our code would not compile with Hadoop v1 libraries because we have several YARN features. However, the code is structured in a way such that we could still run on Hadoop v1 because we would never load the related classes during runtime. To be honest though, we did not run SystemML on v1 for more than a year now. @Deron: In order to answer your question: it would be great if you could update MRConfigurationNames for now - it's anyway good to have all configurations in a central place. Down the road we will systematically remove v1 support because it would allow us to simplify a couple of things. Regards, Matthias From: Luciano Resende <luckbr1...@gmail.com> To: dev@systemml.incubator.apache.org Date: 01/20/2016 01:14 PM Subject: Re: SystemML Hadoop version support On Wed, Jan 20, 2016 at 12:50 PM, Deron Eriksson <deroneriks...@gmail.com> wrote: > Does SystemML still support Hadoop 1? > > I see in MRConfigurationNames a static initialization block for Hadoop > v1/v2 properties. I'd like to remove the current deprecated property > warnings (log messages in the console) that we get with Hadoop v2.4.1 and I > was wondering if I should add the old v1 and new v2 versions to > MRConfigurationNames or if I can just add the new v2 versions and the v1 > versions can be removed. > > If we still support Hadoop v1, we might want to update the pom with > profiles for Hadoop 1 and 2. > > Deron > Even Spark is moving away of supporting some old versions of Hadoop, etc... I would really like us to be moving kind of together with other dependencies, in the sense of supporting Hadoop 2.6 and up, and Spark 1.6 and maybe 1.5.2 as well. -- Luciano Resende http://people.apache.org/~lresende http://twitter.com/lresende1975 http://lresende.blogspot.com/
