Issue during Spark streaming with ZeroMQ source

2014-04-29 Thread Francis . Hu
Hi, all

 

I installed spark-0.9.1 and zeromq 4.0.1 , and then run below example:

 

./bin/run-example org.apache.spark.streaming.examples.SimpleZeroMQPublisher
tcp://127.0.1.1:1234 foo.bar`

./bin/run-example org.apache.spark.streaming.examples.ZeroMQWordCount
local[2] tcp://127.0.1.1:1234 foo`

 

No any message was received in ZeroMQWordCount side. 

 

Does anyone know what the issue is ? 

 

 

Thanks,

Francis

 



Re: Issue during Spark streaming with ZeroMQ source

2014-04-29 Thread Prashant Sharma
Unfortunately zeromq 4.0.1 is not supported.
https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/streaming/examples/ZeroMQWordCount.scala#L63Says
about the version. You will need that version of zeromq to see it
work. Basically I have seen it working nicely with zeromq 2.2.0 and if you
have jzmq libraries installed performance is much better.

Prashant Sharma


On Tue, Apr 29, 2014 at 12:29 PM, Francis.Hu
francis...@reachjunction.comwrote:

  Hi, all



 I installed spark-0.9.1 and zeromq 4.0.1 , and then run below example:



 ./bin/run-example
 org.apache.spark.streaming.examples.SimpleZeroMQPublisher *tcp*://
 127.0.1.1:1234 foo.bar`

 ./bin/run-example org.apache.spark.streaming.examples.ZeroMQWordCount
 local[2] *tcp*://127.0.1.1:1234 *foo*`



 No any message was received in ZeroMQWordCount side.



 Does anyone know what the issue is ?





 Thanks,

 Francis





Re: 答复: Issue during Spark streaming with ZeroMQ source

2014-04-29 Thread Prashant Sharma
Well that is not going to be easy, simply because we depend on akka-zeromq
for zeromq support. And since akka does not support the latest zeromq
library yet, I doubt if there is something simple that can be done to
support it.

Prashant Sharma


On Tue, Apr 29, 2014 at 2:44 PM, Francis.Hu francis...@reachjunction.comwrote:

  Thanks, Prashant Sharma





 It works right now after degrade zeromq from 4.0.1 to  2.2.

 Do you know the new release of spark  whether it will upgrade zeromq ?

 Many of our programs are using zeromq 4.0.1, so if in next release ,spark
 streaming can release with a newer zeromq  that would be better for us.





 Francis.



 *发件人:* Prashant Sharma [mailto:scrapco...@gmail.com]
 *发送时间:* Tuesday, April 29, 2014 15:53
 *收件人:* user@spark.apache.org
 *主题:* Re: Issue during Spark streaming with ZeroMQ source



 Unfortunately zeromq 4.0.1 is not supported.
 https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/streaming/examples/ZeroMQWordCount.scala#L63Says
  about the version. You will need that version of zeromq to see it
 work. Basically I have seen it working nicely with zeromq 2.2.0 and if you
 have jzmq libraries installed performance is much better.


   Prashant Sharma



 On Tue, Apr 29, 2014 at 12:29 PM, Francis.Hu francis...@reachjunction.com
 wrote:

 Hi, all



 I installed spark-0.9.1 and zeromq 4.0.1 , and then run below example:



 ./bin/run-example
 org.apache.spark.streaming.examples.SimpleZeroMQPublisher *tcp*://
 127.0.1.1:1234 foo.bar`

 ./bin/run-example org.apache.spark.streaming.examples.ZeroMQWordCount
 local[2] *tcp*://127.0.1.1:1234 *foo*`



 No any message was received in ZeroMQWordCount side.



 Does anyone know what the issue is ?





 Thanks,

 Francis