Hi,
It could be that the timestamp of the file is old. Moving the file does not
update the file's timestamp. After you have launched the job, either
'touch' the file if it's already in /opt/test/ to update the timestamp or
'cp' the file to a temporary directory then 'mv' it to /opt/test/.
HTH,
Deng
On Sat, Nov 14, 2015 at 9:51 AM, ravi.gawai wrote:
> Hi,
> I am trying simple file streaming example using
> Sparkstreaming(spark-streaming_2.10,version:1.5.1)
>
> public class DStreamExample {
>
> public static void main(final String[] args) {
>
>final SparkConf sparkConf = new SparkConf();
> sparkConf.setAppName("SparkJob");
>
> sparkConf.setMaster("local[4]"); // for local
>
> final JavaSparkContext sc = new JavaSparkContext(sparkConf);
>
> final JavaStreamingContext ssc = new JavaStreamingContext(sc,
> new Duration(2000));
>
> final JavaDStream lines = ssc.textFileStream("/opt/test/");
>
> lines.print();
>
> ssc.start();
> ssc.awaitTermination();
> }
> }
>
> When I run this code on single file or director it does not print anything
> from file, I see in logs its constantly polling but nothing is printed. I
> tried moving file to directory when this program was running.
>
> Is there something I am missing? I tried applying map function on lines
> RDD
> that also does not work.
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spak-filestreaming-issue-tp25380.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
