RE: Problem connecting to HDFS in Spark shell

2014-04-21 Thread Williams, Ken
I figured it out - I should be using textFile(...), not hadoopFile(...). And my HDFS URL should include the host: hdfs://host/user/kwilliams/corTable2/part-m-0 I haven't figured out how to let the hostname default to the host mentioned in our /etc/hadoop/conf/hdfs-site.xml like the

Re: Problem connecting to HDFS in Spark shell

2014-04-21 Thread Marcelo Vanzin
Hi Ken, On Mon, Apr 21, 2014 at 1:39 PM, Williams, Ken ken.willi...@windlogics.com wrote: I haven't figured out how to let the hostname default to the host mentioned in our /etc/hadoop/conf/hdfs-site.xml like the Hadoop command-line tools do, but that's not so important. Try adding

RE: Problem connecting to HDFS in Spark shell

2014-04-21 Thread Williams, Ken
-Original Message- From: Marcelo Vanzin [mailto:van...@cloudera.com] Hi Ken, On Mon, Apr 21, 2014 at 1:39 PM, Williams, Ken ken.willi...@windlogics.com wrote: I haven't figured out how to let the hostname default to the host mentioned in our /etc/hadoop/conf/hdfs-site.xml like the