have a particular concern or use case that
relies on ordering between A, B and X2?
-adrian
From: Nipun Arora
Date: Sunday, October 25, 2015 at 4:09 PM
To: Andy Dang
Cc: user
Subject: Re: [SPARK STREAMING] Concurrent operations in spark streaming
So essentially the driver/client program needs
Tanase
Date: Monday, October 26, 2015 at 11:57 AM
To: Nipun Arora, Andy Dang
Cc: user
Subject: Re: [SPARK STREAMING] Concurrent operations in spark streaming
If I understand the order correctly, not really. First of all, the easiest way
to make sure it works as expected is to check out the visual DAG
I wanted to understand something about the internals of spark streaming
executions.
If I have a stream X, and in my program I send stream X to function A and
function B:
1. In function A, I do a few transform/filter operations etc. on X->Y->Z to
create stream Z. Now I do a forEach Operation on Z
If you execute the collect step (foreach in 1, possibly reduce in 2) in two
threads in the driver then both of them will be executed in parallel.
Whichever gets submitted to Spark first gets executed first - you can use a
semaphore if you need to ensure the ordering of execution, though I would
