On Wed, Dec 9, 2015 at 4:29 PM, Adrian Bridgett <adr...@opensignal.com> wrote:
> (resending, text only as first post on 2nd never seemed to make it) > > Using parallelize() on a dataset I'm only seeing two tasks rather than the > number of cores in the Mesos cluster. This is with spark 1.5.1 and using > the mesos coarse grained scheduler. > > Running pyspark in a console seems to show that it's taking a while before > the mesos executors come online (at which point the default parallelism is > changing). If I add "sleep 30" after initialising the SparkContext I get > the "right" number (42 by coincidence!) > > I've just tried increasing minRegisteredResourcesRatio to 0.5 but this > doesn't affect either the test case below nor my code. > This limit seems to be implemented only in the coarse-grained Mesos scheduler, but the fix will be available starting with Spark 1.6.0 (1.5.2 doesn't have it). iulian > > Is there something else I can do instead? Perhaps it should be seeing how > many tasks _should_ be available rather than how many are (I'm also using > dynamicAllocation). > > 15/12/02 14:34:09 INFO mesos.CoarseMesosSchedulerBackend: SchedulerBackend > is ready for scheduling beginning after reached > minRegisteredResourcesRatio: 0.0 > >>> > >>> > >>> print (sc.defaultParallelism) > 2 > >>> 15/12/02 14:34:12 INFO mesos.CoarseMesosSchedulerBackend: Mesos task 5 > is now TASK_RUNNING > 15/12/02 14:34:13 INFO mesos.MesosExternalShuffleClient: Successfully > registered app 20151117-115458-164233482-5050-24333-0126 with external > shuffle service. > .... > 15/12/02 14:34:15 INFO mesos.CoarseMesosSchedulerBackend: Registered > executor: > AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@ip-10-1-200-147.ec2.internal:41194/user/Executor#-1021429650]) > with ID 20151117-115458-164233482-5050-24333-S22/5 > 15/12/02 14:34:15 INFO spark.ExecutorAllocationManager: New executor > 20151117-115458-164233482-5050-24333-S22/5 has registered (new total is 1) > .... > >>> print (sc.defaultParallelism) > 42 > > Thanks > > Adrian Bridgett > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > > -- -- Iulian Dragos ------ Reactive Apps on the JVM www.typesafe.com
