Can you paste your complete spark-submit command? Also did you try
specifying *--worker-cores*?

Thanks
Best Regards

On Tue, Mar 10, 2015 at 9:00 PM, htailor <[email protected]> wrote:

> Hi All,
>
> I need some help with a problem in pyspark which is causing a major issue.
>
> Recently I've noticed that the behaviour of the python.deamons on the
> worker
> nodes for compute-intensive tasks have changed from using all the avaliable
> cores to using only a single core. On each worker node, 8 python.deamons
> exist but they all seem to run on a single core. The remaining 7 cores
> idle.
>
> Our hardware consists of 9 hosts (1x driver node and 8x worker nodes) each
> with 8 cores and 64gb RAM - we are using Spark 1.2.0-SNAPSHOT (Python) in
> standalone mode via Cloudera 5.3.2.
>
> To give a better understanding of the problem I have made a quick script
> from one of the given examples which replicates the problem:
>
> When I run the calculating_pies.py script using the command "spark-submit
> calculate_pies.py" this is what I typically see on all my worker nodes:
>
>   1  [|||||||||||||||||||||||||||||||||||||||||||||||||||100.0%]      5
> [|||                                                  3.9%]
>   2  [
> 0.0%]      6  [|||                                                  2.6%]
>   3  [||
> 1.3%]     7  [|||||                                                8.4%]
>   4  [||
> 1.3%]     8  [||||||                                               8.7%]
>
>   PID USER      PRI  NI  VIRT   RES    SHR S CPU% MEM%   TIME+  Command
> 30672 spark    20   0   225M  112M  1156 R 13.0   0.2       0:03.10 python
> -m pyspark.daemon
> 30681 spark    20   0   225M  112M  1152 R 13.0   0.2       0:03.10 python
> -m pyspark.daemon
> 30687 spark    20   0   225M  112M  1152 R 13.0   0.2       0:03.10 python
> -m pyspark.daemon
> 30678 spark    20   0   225M  112M  1152 R 12.0   0.2       0:03.10 python
> -m pyspark.daemon
> 30693 spark    20   0   225M  112M  1152 R 12.0   0.2       0:03.08 python
> -m pyspark.daemon
> 30674 spark    20   0   225M  112M  1152 R 12.0   0.2       0:03.10 python
> -m pyspark.daemon
> 30688 spark    20   0   225M  112M  1152 R 12.0   0.2       0:03.08 python
> -m pyspark.daemon
> 30684 spark    20   0   225M  112M  1152 R 12.0   0.2       0:03.10 python
> -m pyspark.daemon
>
> Through the spark UI I do see 8 executor ids with 8 active tasks on each. I
> also see the same behaviour if I use the flag --total-executor-cores 64 in
> spark-submit.
>
> Strangly, If I run the same script in local mode everything seems to run
> fine. This is what I see
>
>   1  [||||||||||||||||||||||||||||||||||||||||||||||100.0%]     5
> [||||||||||||||||||||||||||||||||||||||||||||||100.0%]
>   2  [||||||||||||||||||||||||||||||||||||||||||||||100.0%]     6
> [||||||||||||||||||||||||||||||||||||||||||||||100.0%]
>   3  [||||||||||||||||||||||||||||||||||||||||||||||100.0%]     7
> [||||||||||||||||||||||||||||||||||||||||||||||100.0%]
>   4  [|||||||||||||||||||||||||||||||||||||||||||||||99.3%]     8
> [|||||||||||||||||||||||||||||||||||||||||||||||99.4%]
>
>   PID USER      PRI  NI  VIRT   RES    SHR  S CPU% MEM%   TIME+  Command
> 22519 data       20   0  225M  106M  1368 R 99.0   0.2        0:10.97
> python
> -m pyspark.daemon
> 22508 data       20   0  225M  106M  1368 R 99.0   0.2        0:10.92
> python
> -m pyspark.daemon
> 22513 data       20   0  225M  106M  1368 R 99.0   0.2        0:11.02
> python
> -m pyspark.daemon
> 22526 data       20   0  225M  106M  1368 R 99.0   0.2        0:10.84
> python
> -m pyspark.daemon
> 22522 data       20   0  225M  106M  1368 R 98.0   0.2        0:10.95
> python
> -m pyspark.daemon
> 22523 data       20   0  225M  106M  1368 R 97.0   0.2        0:10.92
> python
> -m pyspark.daemon
> 22507 data       20   0  225M  106M  1368 R 97.0   0.2        0:10.83
> python
> -m pyspark.daemon
> 22516 data       20   0  225M  106M  1368 R 93.0   0.2        0:10.88
> python
> -m pyspark.daemon
>
>
> ============== calculating_pies.py ==================
>
> #!/usr/bin/pyspark
>
> import random
> from pyspark.conf import SparkConf
> from pyspark.context import SparkContext
> from pyspark.storagelevel import StorageLevel
>
> def pi(NUM_SAMPLE = 1000000):
>     count = 0.
>     for i in xrange(NUM_SAMPLE):
>         x, y = random.random(), random.random()
>         if (x * x) + (y * y) < 1:
>             count += 1
>     return 4.0 * (count / NUM_SAMPLE)
>
> if __name__ == "__main__":
>
>     sconf = (SparkConf()
>              .set('spark.default.parallelism','256')
>              .set('spark.app.name', 'Calculating PI'))
>
>     # local
>     # sc = SparkContext(conf=sconf)
>
>     # standalone
>     sc = SparkContext("spark://<driver_host>:7077", conf=sconf)
>
>     # yarn
>     # sc = SparkContext("yarn-client", conf=sconf)
>
>     rdd_pies = sc.parallelize(range(10000), 1000)
>     rdd_pies.map(lambda x: pi()).collect()
>     sc.stop()
>
> =====================================================
>
> Does anyone have any suggestions, or know of any config we should be
> looking
> at that could solve this problem? Does anyone else see the same problem?
>
> Any help is appreciated, Thanks.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Pyspark-not-using-all-cores-tp21989.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>
>

Reply via email to