Instead of sending the results of the one spark app directly to the other
one, you could write the results to a Kafka topic which is consumed by your
other spark application.
On Fri, Nov 20, 2015 at 12:07 PM Saiph Kappa wrote:
> I think my problem persists whether I use Kafka or sockets. Or am I
You're confused about which parts of your code are running on the driver vs
the executor, which is why you're getting serialization errors.
Read
http://spark.apache.org/docs/latest/streaming-programming-guide.html#design-patterns-for-using-foreachrdd
On Fri, Nov 20, 2015 at 1:07 PM, Saiph Kapp
I think my problem persists whether I use Kafka or sockets. Or am I wrong?
How would you use Kafka here?
On Fri, Nov 20, 2015 at 7:12 PM, Christian wrote:
> Have you considered using Kafka?
>
> On Fri, Nov 20, 2015 at 6:48 AM Saiph Kappa wrote:
>
>> Hi,
>>
>> I have a basic spark streaming appl
Have you considered using Kafka?
On Fri, Nov 20, 2015 at 6:48 AM Saiph Kappa wrote:
> Hi,
>
> I have a basic spark streaming application like this:
>
> «
> ...
>
> val ssc = new StreamingContext(sparkConf, Duration(batchMillis))
> val rawStreams = (1 to numStreams).map(_ =>
> ssc.rawSocketStrea
Hi,
I have a basic spark streaming application like this:
«
...
val ssc = new StreamingContext(sparkConf, Duration(batchMillis))
val rawStreams = (1 to numStreams).map(_ =>
ssc.rawSocketStream[String](host, port, StorageLevel.MEMORY_ONLY_SER)).toArray
val union = ssc.union(rawStreams)
union.f
