Re: Re: spark streaming printing no output

2015-04-16 Thread jay vyas 
empty folders generally means that you need to just increase the window
intervals; i.e. spark streaming
saveAsTxtFiles will save folders for each interval regardless

On Wed, Apr 15, 2015 at 5:03 AM, Shushant Arora shushantaror...@gmail.com
wrote:

 Its printing on console but on HDFS all folders are still empty .

 On Wed, Apr 15, 2015 at 2:29 PM, Shushant Arora shushantaror...@gmail.com
  wrote:

 Thanks !! Yes message types on this console is seen on another console.

 When I closed another console, spark streaming job is printing messages
 on console .

  Isn't the message written on a port using netcat be avaible for multiple
 consumers?

 On Wed, Apr 15, 2015 at 2:22 PM, bit1...@163.com bit1...@163.com wrote:

 Looks  the message is consumed by the another console?( can see
 messages typed on this port from another console.)

 --
 bit1...@163.com


 *From:* Shushant Arora shushantaror...@gmail.com
 *Date:* 2015-04-15 17:11
 *To:* Akhil Das ak...@sigmoidanalytics.com
 *CC:* user@spark.apache.org
 *Subject:* Re: spark streaming printing no output
 When I launched spark-shell using, spark-shell ---master local[2].
 Same behaviour, no output on console but only timestamps.

 When I did, lines.saveAsTextFiles(hdfslocation,suffix);
 I get empty files of 0 bytes on hdfs

 On Wed, Apr 15, 2015 at 12:46 PM, Akhil Das ak...@sigmoidanalytics.com
 wrote:

 Just make sure you have atleast 2 cores available for processing. You
 can try launching it in local[2] and make sure its working fine.

 Thanks
 Best Regards

 On Tue, Apr 14, 2015 at 11:41 PM, Shushant Arora 
 shushantaror...@gmail.com wrote:

 Hi

 I am running a spark streaming application but on console nothing is
 getting printed.

 I am doing
 1.bin/spark-shell --master clusterMgrUrl
 2.import org.apache.spark.streaming.StreamingContext
 import org.apache.spark.streaming.StreamingContext._
 import org.apache.spark.streaming.dstream.DStream
 import org.apache.spark.streaming.Duration
 import org.apache.spark.streaming.Seconds
 val ssc = new StreamingContext( sc, Seconds(1))
 val lines = ssc.socketTextStream(hostname,)
 lines.print()
 ssc.start()
 ssc.awaitTermination()

 Jobs are getting created when I see webUI but nothing gets printed on
 console.

 I have started a nc script on hostname  port  and can see messages
 typed on this port from another console.



 Please let me know If I am doing something wrong.










-- 
jay vyas

Re: spark streaming printing no output

2015-04-15 Thread Shushant Arora 
Yes only Time: 142905487 ms  strings gets printed on console.
No output is getting printed.
And timeinterval between two strings of form ( time:ms)is very less
than Streaming Duration set in program.

On Wed, Apr 15, 2015 at 5:11 AM, Shixiong Zhu zsxw...@gmail.com wrote:

 Could you see something like this in the console?

 ---
 Time: 142905487 ms
 ---


 Best Regards,
 Shixiong(Ryan) Zhu

 2015-04-15 2:11 GMT+08:00 Shushant Arora shushantaror...@gmail.com:

 Hi

 I am running a spark streaming application but on console nothing is
 getting printed.

 I am doing
 1.bin/spark-shell --master clusterMgrUrl
 2.import org.apache.spark.streaming.StreamingContext
 import org.apache.spark.streaming.StreamingContext._
 import org.apache.spark.streaming.dstream.DStream
 import org.apache.spark.streaming.Duration
 import org.apache.spark.streaming.Seconds
 val ssc = new StreamingContext( sc, Seconds(1))
 val lines = ssc.socketTextStream(hostname,)
 lines.print()
 ssc.start()
 ssc.awaitTermination()

 Jobs are getting created when I see webUI but nothing gets printed on
 console.

 I have started a nc script on hostname  port  and can see messages
 typed on this port from another console.



 Please let me know If I am doing something wrong.

Re: spark streaming printing no output

2015-04-15 Thread Akhil Das 
Just make sure you have atleast 2 cores available for processing. You can
try launching it in local[2] and make sure its working fine.

Thanks
Best Regards

On Tue, Apr 14, 2015 at 11:41 PM, Shushant Arora shushantaror...@gmail.com
wrote:

 Hi

 I am running a spark streaming application but on console nothing is
 getting printed.

 I am doing
 1.bin/spark-shell --master clusterMgrUrl
 2.import org.apache.spark.streaming.StreamingContext
 import org.apache.spark.streaming.StreamingContext._
 import org.apache.spark.streaming.dstream.DStream
 import org.apache.spark.streaming.Duration
 import org.apache.spark.streaming.Seconds
 val ssc = new StreamingContext( sc, Seconds(1))
 val lines = ssc.socketTextStream(hostname,)
 lines.print()
 ssc.start()
 ssc.awaitTermination()

 Jobs are getting created when I see webUI but nothing gets printed on
 console.

 I have started a nc script on hostname  port  and can see messages
 typed on this port from another console.



 Please let me know If I am doing something wrong.

Re: spark streaming printing no output

2015-04-15 Thread Shixiong Zhu 
So the time niterval is much less than 1000 ms as you set in the code?
That's weird. Could you check the whole outputs to confirm that the content
won't be flushed by logs?

Best Regards,
Shixiong(Ryan) Zhu

2015-04-15 15:04 GMT+08:00 Shushant Arora shushantaror...@gmail.com:

 Yes only Time: 142905487 ms  strings gets printed on console.
 No output is getting printed.
 And timeinterval between two strings of form ( time:ms)is very less
 than Streaming Duration set in program.

 On Wed, Apr 15, 2015 at 5:11 AM, Shixiong Zhu zsxw...@gmail.com wrote:

 Could you see something like this in the console?

 ---
 Time: 142905487 ms
 ---


 Best Regards,
 Shixiong(Ryan) Zhu

 2015-04-15 2:11 GMT+08:00 Shushant Arora shushantaror...@gmail.com:

 Hi

 I am running a spark streaming application but on console nothing is
 getting printed.

 I am doing
 1.bin/spark-shell --master clusterMgrUrl
 2.import org.apache.spark.streaming.StreamingContext
 import org.apache.spark.streaming.StreamingContext._
 import org.apache.spark.streaming.dstream.DStream
 import org.apache.spark.streaming.Duration
 import org.apache.spark.streaming.Seconds
 val ssc = new StreamingContext( sc, Seconds(1))
 val lines = ssc.socketTextStream(hostname,)
 lines.print()
 ssc.start()
 ssc.awaitTermination()

 Jobs are getting created when I see webUI but nothing gets printed on
 console.

 I have started a nc script on hostname  port  and can see messages
 typed on this port from another console.



 Please let me know If I am doing something wrong.

Re: spark streaming printing no output

2015-04-15 Thread Shushant Arora 
When I launched spark-shell using, spark-shell ---master local[2].
Same behaviour, no output on console but only timestamps.

When I did, lines.saveAsTextFiles(hdfslocation,suffix);
I get empty files of 0 bytes on hdfs

On Wed, Apr 15, 2015 at 12:46 PM, Akhil Das ak...@sigmoidanalytics.com
wrote:

 Just make sure you have atleast 2 cores available for processing. You can
 try launching it in local[2] and make sure its working fine.

 Thanks
 Best Regards

 On Tue, Apr 14, 2015 at 11:41 PM, Shushant Arora 
 shushantaror...@gmail.com wrote:

 Hi

 I am running a spark streaming application but on console nothing is
 getting printed.

 I am doing
 1.bin/spark-shell --master clusterMgrUrl
 2.import org.apache.spark.streaming.StreamingContext
 import org.apache.spark.streaming.StreamingContext._
 import org.apache.spark.streaming.dstream.DStream
 import org.apache.spark.streaming.Duration
 import org.apache.spark.streaming.Seconds
 val ssc = new StreamingContext( sc, Seconds(1))
 val lines = ssc.socketTextStream(hostname,)
 lines.print()
 ssc.start()
 ssc.awaitTermination()

 Jobs are getting created when I see webUI but nothing gets printed on
 console.

 I have started a nc script on hostname  port  and can see messages
 typed on this port from another console.



 Please let me know If I am doing something wrong.

Re: Re: spark streaming printing no output

2015-04-15 Thread bit1...@163.com 
Looks  the message is consumed by the another console?( can see messages typed 
on this port from another console.)



bit1...@163.com
 
From: Shushant Arora
Date: 2015-04-15 17:11
To: Akhil Das
CC: user@spark.apache.org
Subject: Re: spark streaming printing no output
When I launched spark-shell using, spark-shell ---master local[2].
Same behaviour, no output on console but only timestamps.

When I did, lines.saveAsTextFiles(hdfslocation,suffix);
I get empty files of 0 bytes on hdfs 

On Wed, Apr 15, 2015 at 12:46 PM, Akhil Das ak...@sigmoidanalytics.com wrote:
Just make sure you have atleast 2 cores available for processing. You can try 
launching it in local[2] and make sure its working fine.

Thanks
Best Regards

On Tue, Apr 14, 2015 at 11:41 PM, Shushant Arora shushantaror...@gmail.com 
wrote:
Hi 

I am running a spark streaming application but on console nothing is getting 
printed.

I am doing
1.bin/spark-shell --master clusterMgrUrl
2.import org.apache.spark.streaming.StreamingContext 
import org.apache.spark.streaming.StreamingContext._ 
import org.apache.spark.streaming.dstream.DStream
import org.apache.spark.streaming.Duration
import org.apache.spark.streaming.Seconds
val ssc = new StreamingContext( sc, Seconds(1))
val lines = ssc.socketTextStream(hostname,)
lines.print()
ssc.start()
ssc.awaitTermination()

Jobs are getting created when I see webUI but nothing gets printed on console.

I have started a nc script on hostname  port  and can see messages typed on 
this port from another console.



Please let me know If I am doing something wrong.

Re: Re: spark streaming printing no output

2015-04-15 Thread Shushant Arora 
Its printing on console but on HDFS all folders are still empty .

On Wed, Apr 15, 2015 at 2:29 PM, Shushant Arora shushantaror...@gmail.com
wrote:

 Thanks !! Yes message types on this console is seen on another console.

 When I closed another console, spark streaming job is printing messages on
 console .

  Isn't the message written on a port using netcat be avaible for multiple
 consumers?

 On Wed, Apr 15, 2015 at 2:22 PM, bit1...@163.com bit1...@163.com wrote:

 Looks  the message is consumed by the another console?( can see messages
 typed on this port from another console.)

 --
 bit1...@163.com


 *From:* Shushant Arora shushantaror...@gmail.com
 *Date:* 2015-04-15 17:11
 *To:* Akhil Das ak...@sigmoidanalytics.com
 *CC:* user@spark.apache.org
 *Subject:* Re: spark streaming printing no output
 When I launched spark-shell using, spark-shell ---master local[2].
 Same behaviour, no output on console but only timestamps.

 When I did, lines.saveAsTextFiles(hdfslocation,suffix);
 I get empty files of 0 bytes on hdfs

 On Wed, Apr 15, 2015 at 12:46 PM, Akhil Das ak...@sigmoidanalytics.com
 wrote:

 Just make sure you have atleast 2 cores available for processing. You
 can try launching it in local[2] and make sure its working fine.

 Thanks
 Best Regards

 On Tue, Apr 14, 2015 at 11:41 PM, Shushant Arora 
 shushantaror...@gmail.com wrote:

 Hi

 I am running a spark streaming application but on console nothing is
 getting printed.

 I am doing
 1.bin/spark-shell --master clusterMgrUrl
 2.import org.apache.spark.streaming.StreamingContext
 import org.apache.spark.streaming.StreamingContext._
 import org.apache.spark.streaming.dstream.DStream
 import org.apache.spark.streaming.Duration
 import org.apache.spark.streaming.Seconds
 val ssc = new StreamingContext( sc, Seconds(1))
 val lines = ssc.socketTextStream(hostname,)
 lines.print()
 ssc.start()
 ssc.awaitTermination()

 Jobs are getting created when I see webUI but nothing gets printed on
 console.

 I have started a nc script on hostname  port  and can see messages
 typed on this port from another console.



 Please let me know If I am doing something wrong.

Re: Re: spark streaming printing no output

2015-04-15 Thread Shushant Arora 
Thanks !! Yes message types on this console is seen on another console.

When I closed another console, spark streaming job is printing messages on
console .

 Isn't the message written on a port using netcat be avaible for multiple
consumers?

On Wed, Apr 15, 2015 at 2:22 PM, bit1...@163.com bit1...@163.com wrote:

 Looks  the message is consumed by the another console?( can see messages
 typed on this port from another console.)

 --
 bit1...@163.com


 *From:* Shushant Arora shushantaror...@gmail.com
 *Date:* 2015-04-15 17:11
 *To:* Akhil Das ak...@sigmoidanalytics.com
 *CC:* user@spark.apache.org
 *Subject:* Re: spark streaming printing no output
 When I launched spark-shell using, spark-shell ---master local[2].
 Same behaviour, no output on console but only timestamps.

 When I did, lines.saveAsTextFiles(hdfslocation,suffix);
 I get empty files of 0 bytes on hdfs

 On Wed, Apr 15, 2015 at 12:46 PM, Akhil Das ak...@sigmoidanalytics.com
 wrote:

 Just make sure you have atleast 2 cores available for processing. You can
 try launching it in local[2] and make sure its working fine.

 Thanks
 Best Regards

 On Tue, Apr 14, 2015 at 11:41 PM, Shushant Arora 
 shushantaror...@gmail.com wrote:

 Hi

 I am running a spark streaming application but on console nothing is
 getting printed.

 I am doing
 1.bin/spark-shell --master clusterMgrUrl
 2.import org.apache.spark.streaming.StreamingContext
 import org.apache.spark.streaming.StreamingContext._
 import org.apache.spark.streaming.dstream.DStream
 import org.apache.spark.streaming.Duration
 import org.apache.spark.streaming.Seconds
 val ssc = new StreamingContext( sc, Seconds(1))
 val lines = ssc.socketTextStream(hostname,)
 lines.print()
 ssc.start()
 ssc.awaitTermination()

 Jobs are getting created when I see webUI but nothing gets printed on
 console.

 I have started a nc script on hostname  port  and can see messages
 typed on this port from another console.



 Please let me know If I am doing something wrong.

spark streaming printing no output

2015-04-14 Thread Shushant Arora 
Hi

I am running a spark streaming application but on console nothing is
getting printed.

I am doing
1.bin/spark-shell --master clusterMgrUrl
2.import org.apache.spark.streaming.StreamingContext
import org.apache.spark.streaming.StreamingContext._
import org.apache.spark.streaming.dstream.DStream
import org.apache.spark.streaming.Duration
import org.apache.spark.streaming.Seconds
val ssc = new StreamingContext( sc, Seconds(1))
val lines = ssc.socketTextStream(hostname,)
lines.print()
ssc.start()
ssc.awaitTermination()

Jobs are getting created when I see webUI but nothing gets printed on
console.

I have started a nc script on hostname  port  and can see messages
typed on this port from another console.



Please let me know If I am doing something wrong.

Re: spark streaming printing no output

2015-04-14 Thread Shixiong Zhu 
Could you see something like this in the console?

---
Time: 142905487 ms
---


Best Regards,
Shixiong(Ryan) Zhu

2015-04-15 2:11 GMT+08:00 Shushant Arora shushantaror...@gmail.com:

 Hi

 I am running a spark streaming application but on console nothing is
 getting printed.

 I am doing
 1.bin/spark-shell --master clusterMgrUrl
 2.import org.apache.spark.streaming.StreamingContext
 import org.apache.spark.streaming.StreamingContext._
 import org.apache.spark.streaming.dstream.DStream
 import org.apache.spark.streaming.Duration
 import org.apache.spark.streaming.Seconds
 val ssc = new StreamingContext( sc, Seconds(1))
 val lines = ssc.socketTextStream(hostname,)
 lines.print()
 ssc.start()
 ssc.awaitTermination()

 Jobs are getting created when I see webUI but nothing gets printed on
 console.

 I have started a nc script on hostname  port  and can see messages
 typed on this port from another console.



 Please let me know If I am doing something wrong.