Hey Mike,
Stage ID's are not guaranteed to be sequential because of the way the
DAG scheduler works (only increasing). In some cases stage ID numbers
are skipped when stages are generated.
Any stage/ID that appears in the Spark UI is an actual stage, so if
you see ID's in there, but they are not
Are you seeing the same behavior on the driver UI? (that running on port
4040), If you click on the stage id header you can sort the stages based on
IDs.
Thanks
Best Regards
On Fri, Jun 5, 2015 at 10:21 PM, Mike Hynes 91m...@gmail.com wrote:
Hi folks,
When I look at the output logs for an
Hi folks,
When I look at the output logs for an iterative Spark program, I see
that the stage IDs are not arithmetically numbered---that is, there
are gaps between stages and I might find log information about Stage
0, 1,2, 5, but not 3 or 4.
As an example, the output from the Spark logs below
