True but I need to use the dir tree walking and file list creation code in HDFS 2 in order to create URIs for Spark input. So HDFS 2 it is.
BTW Spark supports more file systems than local: and hdfs:, I assume we are sticking with those two? On Apr 29, 2014, at 11:13 AM, Dmitriy Lyubimov <[email protected]> wrote: I am using cdh 4.3 with hadoop 2.0.0 (as a dependency) (i am not using mr1-cdh.... ) for code that runs over Spark Bindings this is a moot issue since spark can read and write to pretty much any version of hadoop or distro that exists today. One just needs to compile it with proper version of hadoop hdfs/mr/yarn. On Tue, Apr 29, 2014 at 10:51 AM, Pat Ferrel <[email protected]> wrote: > Is there something, a page or Jira that describes the current h2 status? I > think Dmitiry is using h1 now as I am. Is it time to switch? I ask > primarily for access to the filesystem (HDFS or local through hadoop), not > for access to mapreduce jobs. > >
