Resource manager: estimation of application execution time/remaining time.

Dear all, For a particular Spark extension, I would like to know whether it would be possible for the Resource Manager (e.g. Standalone cluster manager) to know or estimate the total execution time of a submitted application, or the execution remaining time of such application. Thanks. --

Spark stage concurrency

Suppose a spark job has two stages with independent dependencies (they do not depend on each other) and they are submitted concurrently/simultaneously (as Tasksets) by the DAG scheduler to the task scheduler. Can someone give more detailed insight on how the cores available on executors are

location of a partition in the cluster/ how parallelize method distribute the RDD partitions over the cluster.

Hi, Any hint about getting the location of a particular RDD partition on the cluster? a workaround? Parallelize method on RDDs partitions the RDD into splits as specified or per as per the default parallelism configuration. Does parallelize actually distribute the partitions into the

Simultaneous spark Jobs execution.

Does Spark handle simulate nous execution of jobs within an application or job execution is blocking i.e. a new job can not be initiated until the previous one commits. What does it mean that : "Spark’s scheduler is fully thread-safe" Thanks. -- View this message in context: