Hi, Thanks for getting back to me. Sorry for the delay. I am still having this issue.
@sun: To clarify, The machine actually has 16 usable threads and the job has more than 100 gzip files. So, there are enough partitions to use all threads. @nicholas: The number of partitions match the number of files: > 100. @Sebastian: I understand the lazy loading behavior. For this reason, I usually use a .count() to force the transformation (.first() will not be enough). Still, during the transformation, only 4 cores are used for processing the input files. I don't know if this issue is noticed by other people. Can anyone reproduce it with v1.1? On Wed, Dec 17, 2014 at 2:14 AM, Nicholas Chammas <[email protected]> wrote: > Rui is correct. > > Check how many partitions your RDD has after loading the gzipped files. e.g. > rdd.getNumPartitions(). > > If that number is way less than the number of cores in your cluster (in your > case I suspect the number is 4), then explicitly repartition the RDD to > match the number of cores in your cluster, or some multiple thereof. > > For example: > > new_rdd = rdd.repartition(sc.defaultParallelism * 3) > > Operations on new_rdd should utilize all the cores in your cluster. > > Nick > > > On Wed Dec 17 2014 at 1:42:16 AM Sun, Rui <[email protected]> wrote: >> >> Gautham, >> >> How many number of gz files do you have? Maybe the reason is that gz file >> is compressed that can't be splitted for processing by Mapreduce. A single >> gz file can only be processed by a single Mapper so that the CPU treads >> can't be fully utilized. >> >> -----Original Message----- >> From: Gautham [mailto:[email protected]] >> Sent: Wednesday, December 10, 2014 3:00 AM >> To: [email protected] >> Subject: pyspark sc.textFile uses only 4 out of 32 threads per node >> >> I am having an issue with pyspark launched in ec2 (using spark-ec2) with 5 >> r3.4xlarge machines where each has 32 threads and 240GB of RAM. When I do >> sc.textFile to load data from a number of gz files, it does not progress as >> fast as expected. When I log-in to a child node and run top, I see only 4 >> threads at 100 cpu. All remaining 28 cores were idle. This is not an issue >> when processing the strings after loading, when all the cores are used to >> process the data. >> >> Please help me with this? What setting can be changed to get the CPU usage >> back up to full? >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/pyspark-sc-textFile-uses-only-4-out-of-32-threads-per-node-tp20595.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] For additional >> commands, e-mail: [email protected] >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> > -- Gautham Anil "The first principle is that you must not fool yourself. And you are the easiest person to fool" - Richard P. Feynman --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
