Try
(hdfs:///localhost:8020/user/data/*)
With 3 /.
Thx
tri
-Original Message-
From: Benjamin Cuthbert [mailto:cuthbert@gmail.com]
Sent: Monday, December 01, 2014 4:41 PM
To: user@spark.apache.org
Subject: hdfs streaming context
All,
Is it possible to stream on HDFS directory
Have you tried just passing a path to ssc.textFileStream() ? It
monitors the path for new files by looking at mtime/atime ; all
new/touched files in the time window appear as an rdd in the dstream.
On 1 December 2014 at 14:41, Benjamin Cuthbert cuthbert@gmail.com wrote:
All,
Is it possible
Yes, in fact, that's the only way it works. You need
hdfs://localhost:8020/user/data, I believe.
(No it's not correct to write hdfs:///...)
On Mon, Dec 1, 2014 at 10:41 PM, Benjamin Cuthbert
cuthbert@gmail.com wrote:
All,
Is it possible to stream on HDFS directory and listen for multiple
Thanks Sean,
That worked just removing the /* and leaving it as /user/data
Seems to be streaming in.
On 1 Dec 2014, at 22:50, Sean Owen so...@cloudera.com wrote:
Yes, in fact, that's the only way it works. You need
hdfs://localhost:8020/user/data, I believe.
(No it's not correct to
@spark.apache.org
Subject: Re: hdfs streaming context
Yes, in fact, that's the only way it works. You need
hdfs://localhost:8020/user/data, I believe.
(No it's not correct to write hdfs:///...)
On Mon, Dec 1, 2014 at 10:41 PM, Benjamin Cuthbert cuthbert@gmail.com
wrote:
All,
Is it possible
Yes but you can't follow three slashes with host:port. No host
probably defaults to whatever is found in your HDFS config.
On Mon, Dec 1, 2014 at 11:02 PM, Bui, Tri tri@verizonwireless.com wrote:
For the streaming example I am working on, Its accepted (hdfs:///user/data)
without the
@spark.apache.org
Subject: Re: hdfs streaming context
Yes but you can't follow three slashes with host:port. No host probably
defaults to whatever is found in your HDFS config.
On Mon, Dec 1, 2014 at 11:02 PM, Bui, Tri tri@verizonwireless.com wrote:
For the streaming example I am working on, Its
