Thx Gene! But my concern is with CPU usage not memory. I want to see if there is anyway to control the number of pyspark.daemon processes that get spawned. We have some restriction on number of CPU's we can use on a node, and number of pyspark.daemon processes that get created dont seem to honor spark.executor.cores property setting
Thanks! On Wed, Jun 15, 2016 at 1:53 PM, Gene Pang <gene.p...@gmail.com> wrote: > As Sven mentioned, you can use Alluxio to store RDDs in off-heap memory, > and you can then share that RDD across different jobs. If you would like to > run Spark on Alluxio, this documentation can help: > http://www.alluxio.org/documentation/master/en/Running-Spark-on-Alluxio.html > > Thanks, > Gene > > On Tue, Jun 14, 2016 at 12:44 AM, agateaaa <agate...@gmail.com> wrote: > >> Hi, >> >> I am seeing this issue too with pyspark (Using Spark 1.6.1). I have set >> spark.executor.cores to 1, but I see that whenever streaming batch starts >> processing data, see python -m pyspark.daemon processes increase gradually >> to about 5, (increasing CPU% on a box about 4-5 times, each pyspark.daemon >> takes up around 100 % CPU) >> >> After the processing is done 4 pyspark.daemon processes go away and we >> are left with one till the next batch run. Also sometimes the CPU usage >> for executor process spikes to about 800% even though spark.executor.core >> is set to 1 >> >> e.g. top output >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> 19634 spark 20 0 8871420 1.790g 32056 S 814.1 2.9 0:39.33 >> /usr/lib/j+ <--EXECUTOR >> >> 13897 spark 20 0 46576 17916 6720 S 100.0 0.0 0:00.17 >> python -m + <--pyspark.daemon >> 13991 spark 20 0 46524 15572 4124 S 98.0 0.0 0:08.18 >> python -m + <--pyspark.daemon >> 14488 spark 20 0 46524 15636 4188 S 98.0 0.0 0:07.25 >> python -m + <--pyspark.daemon >> 14514 spark 20 0 46524 15636 4188 S 94.0 0.0 0:06.72 >> python -m + <--pyspark.daemon >> 14526 spark 20 0 48200 17172 4092 S 0.0 0.0 0:00.38 >> python -m + <--pyspark.daemon >> >> >> >> Is there any way to control the number of pyspark.daemon processes that >> get spawned ? >> >> Thank you >> Agateaaa >> >> On Sun, Mar 27, 2016 at 1:08 AM, Sven Krasser <kras...@gmail.com> wrote: >> >>> Hey Ken, >>> >>> 1. You're correct, cached RDDs live on the JVM heap. (There's an >>> off-heap storage option using Alluxio, formerly Tachyon, with which I have >>> no experience however.) >>> >>> 2. The worker memory setting is not a hard maximum unfortunately. What >>> happens is that during aggregation the Python daemon will check its process >>> size. If the size is larger than this setting, it will start spilling to >>> disk. I've seen many occasions where my daemons grew larger. Also, you're >>> relying on Python's memory management to free up space again once objects >>> are evicted. In practice, leave this setting reasonably small but make sure >>> there's enough free memory on the machine so you don't run into OOM >>> conditions. If the lower memory setting causes strains for your users, make >>> sure they increase the parallelism of their jobs (smaller partitions >>> meaning less data is processed at a time). >>> >>> 3. I believe that is the behavior you can expect when setting >>> spark.executor.cores. I've not experimented much with it and haven't looked >>> at that part of the code, but what you describe also reflects my >>> understanding. Please share your findings here, I'm sure those will be very >>> helpful to others, too. >>> >>> One more suggestion for your users is to move to the Pyspark DataFrame >>> API. Much of the processing will then happen in the JVM, and you will bump >>> into fewer Python resource contention issues. >>> >>> Best, >>> -Sven >>> >>> >>> On Sat, Mar 26, 2016 at 1:38 PM, Carlile, Ken <carli...@janelia.hhmi.org >>> > wrote: >>> >>>> This is extremely helpful! >>>> >>>> I’ll have to talk to my users about how the python memory limit should >>>> be adjusted and what their expectations are. I’m fairly certain we bumped >>>> it up in the dark past when jobs were failing because of insufficient >>>> memory for the python processes. >>>> >>>> So just to make sure I’m understanding correctly: >>>> >>>> >>>> - JVM memory (set by SPARK_EXECUTOR_MEMORY and/or >>>> SPARK_WORKER_MEMORY?) is where the RDDs are stored. Currently both of >>>> those >>>> values are set to 90GB >>>> - spark.python.worker.memory controls how much RAM each python task >>>> can take maximum (roughly speaking. Currently set to 4GB >>>> - spark.task.cpus controls how many java worker threads will exist >>>> and thus indirectly how many pyspark daemon processes will exist >>>> >>>> >>>> I’m also looking into fixing my cron jobs so they don’t stack up by >>>> implementing flock in the jobs and changing how teardowns of the spark >>>> cluster work as far as failed workers. >>>> >>>> Thanks again, >>>> —Ken >>>> >>>> On Mar 26, 2016, at 4:08 PM, Sven Krasser <kras...@gmail.com> wrote: >>>> >>>> My understanding is that the spark.executor.cores setting controls the >>>> number of worker threads in the executor in the JVM. Each worker thread >>>> communicates then with a pyspark daemon process (these are not threads) to >>>> stream data into Python. There should be one daemon process per worker >>>> thread (but as I mentioned I sometimes see a low multiple). >>>> >>>> Your 4GB limit for Python is fairly high, that means even for 12 >>>> workers you're looking at a max of 48GB (and it goes frequently beyond >>>> that). You will be better off using a lower number there and instead >>>> increasing the parallelism of your job (i.e. dividing the job into more and >>>> smaller partitions). >>>> >>>> On Sat, Mar 26, 2016 at 7:10 AM, Carlile, Ken < >>>> carli...@janelia.hhmi.org> wrote: >>>> >>>>> Thanks, Sven! >>>>> >>>>> I know that I’ve messed up the memory allocation, but I’m trying not >>>>> to think too much about that (because I’ve advertised it to my users as >>>>> “90GB for Spark works!” and that’s how it displays in the Spark UI >>>>> (totally >>>>> ignoring the python processes). So I’ll need to deal with that at some >>>>> point… esp since I’ve set the max python memory usage to 4GB to work >>>>> around >>>>> other issues! >>>>> >>>>> The load issue comes in because we have a lot of background cron jobs >>>>> (mostly to clean up after spark…), and those will stack up behind the high >>>>> load and keep stacking until the whole thing comes crashing down. I will >>>>> look into how to avoid this stacking, as I think one of my predecessors >>>>> had >>>>> a way, but that’s why the high load nukes the nodes. I don’t have the >>>>> spark.executor.cores set, but will setting that to say, 12 limit the >>>>> pyspark threads, or will it just limit the jvm threads? >>>>> >>>>> Thanks! >>>>> Ken >>>>> >>>>> On Mar 25, 2016, at 9:10 PM, Sven Krasser <kras...@gmail.com> wrote: >>>>> >>>>> Hey Ken, >>>>> >>>>> I also frequently see more pyspark daemons than configured >>>>> concurrency, often it's a low multiple. (There was an issue pre-1.3.0 that >>>>> caused this to be quite a bit higher, so make sure you at least have a >>>>> recent version; see SPARK-5395.) >>>>> >>>>> Each pyspark daemon tries to stay below the configured memory limit >>>>> during aggregation (which is separate from the JVM heap as you note). >>>>> Since >>>>> the number of daemons can be high and the memory limit is per daemon (each >>>>> daemon is actually a process and not a thread and therefore has its own >>>>> memory it tracks against the configured per-worker limit), I found memory >>>>> depletion to be the main source of pyspark problems on larger data sets. >>>>> Also, as Sea already noted the memory limit is not firm and individual >>>>> daemons can grow larger. >>>>> >>>>> With that said, a run queue of 25 on a 16 core machine does not sound >>>>> great but also not awful enough to knock it offline. I suspect something >>>>> else may be going on. If you want to limit the amount of work running >>>>> concurrently, try reducing spark.executor.cores (under normal >>>>> circumstances >>>>> this would leave parts of your resources underutilized). >>>>> >>>>> Hope this helps! >>>>> -Sven >>>>> >>>>> >>>>> On Fri, Mar 25, 2016 at 10:41 AM, Carlile, Ken < >>>>> carli...@janelia.hhmi.org> wrote: >>>>> >>>>>> Further data on this. >>>>>> I’m watching another job right now where there are 16 pyspark.daemon >>>>>> threads, all of which are trying to get a full core (remember, this is a >>>>>> 16 >>>>>> core machine). Unfortunately , the java process actually running the >>>>>> spark >>>>>> worker is trying to take several cores of its own, driving the load up. >>>>>> I’m >>>>>> hoping someone has seen something like this. >>>>>> >>>>>> —Ken >>>>>> >>>>>> On Mar 21, 2016, at 3:07 PM, Carlile, Ken <carli...@janelia.hhmi.org> >>>>>> wrote: >>>>>> >>>>>> No further input on this? I discovered today that the pyspark.daemon >>>>>> threadcount was actually 48, which makes a little more sense (at least >>>>>> it’s >>>>>> a multiple of 16), and it seems to be happening at reduce and collect >>>>>> portions of the code. >>>>>> >>>>>> —Ken >>>>>> >>>>>> On Mar 17, 2016, at 10:51 AM, Carlile, Ken <carli...@janelia.hhmi.org> >>>>>> wrote: >>>>>> >>>>>> Thanks! I found that part just after I sent the email… whoops. I’m >>>>>> guessing that’s not an issue for my users, since it’s been set that way >>>>>> for >>>>>> a couple of years now. >>>>>> >>>>>> The thread count is definitely an issue, though, since if enough >>>>>> nodes go down, they can’t schedule their spark clusters. >>>>>> >>>>>> —Ken >>>>>> >>>>>> On Mar 17, 2016, at 10:50 AM, Ted Yu <yuzhih...@gmail.com> wrote: >>>>>> >>>>>> I took a look at docs/configuration.md >>>>>> Though I didn't find answer for your first question, I think the >>>>>> following pertains to your second question: >>>>>> >>>>>> <tr> >>>>>> <td><code>spark.python.worker.memory</code></td> >>>>>> <td>512m</td> >>>>>> <td> >>>>>> Amount of memory to use per python worker process during >>>>>> aggregation, in the same >>>>>> format as JVM memory strings (e.g. <code>512m</code>, >>>>>> <code>2g</code>). If the memory >>>>>> used during aggregation goes above this amount, it will spill the >>>>>> data into disks. >>>>>> </td> >>>>>> </tr> >>>>>> >>>>>> On Thu, Mar 17, 2016 at 7:43 AM, Carlile, Ken < >>>>>> carli...@janelia.hhmi.org> wrote: >>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> We have an HPC cluster that we run Spark jobs on using standalone >>>>>>> mode and a number of scripts I’ve built up to dynamically schedule and >>>>>>> start spark clusters within the Grid Engine framework. Nodes in the >>>>>>> cluster >>>>>>> have 16 cores and 128GB of RAM. >>>>>>> >>>>>>> My users use pyspark heavily. We’ve been having a number of problems >>>>>>> with nodes going offline with extraordinarily high load. I was able to >>>>>>> look >>>>>>> at one of those nodes today before it went truly sideways, and I >>>>>>> discovered >>>>>>> that the user was running 50 pyspark.daemon threads (remember, this is >>>>>>> a 16 >>>>>>> core box), and the load was somewhere around 25 or so, with all CPUs >>>>>>> maxed >>>>>>> out at 100%. >>>>>>> >>>>>>> So while the spark worker is aware it’s only got 16 cores and >>>>>>> behaves accordingly, pyspark seems to be happy to overrun everything >>>>>>> like >>>>>>> crazy. Is there a global parameter I can use to limit pyspark threads >>>>>>> to a >>>>>>> sane number, say 15 or 16? It would also be interesting to set a memory >>>>>>> limit, which leads to another question. >>>>>>> >>>>>>> How is memory managed when pyspark is used? I have the spark worker >>>>>>> memory set to 90GB, and there is 8GB of system overhead (GPFS caching), >>>>>>> so >>>>>>> if pyspark operates outside of the JVM memory pool, that leaves it at >>>>>>> most >>>>>>> 30GB to play with, assuming there is no overhead outside the JVM’s 90GB >>>>>>> heap (ha ha.) >>>>>>> >>>>>>> Thanks, >>>>>>> Ken Carlile >>>>>>> Sr. Unix Engineer >>>>>>> HHMI/Janelia Research Campus >>>>>>> 571-209-4363 >>>>>>> >>>>>>> >>>>>> >>>>>> Т���������������������������������������������������������������������ХF� >>>>>> V�7V'67&�&R� R�� �â W6W"�V�7V'67&�&T 7 &�� 6�R��&pФf�" FF�F��� � 6��� >>>>>> �G2� >>>>>> R�� �â W6W"ֆV� 7 &�� 6�R��&pР >>>>>> >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For >>>>>> additional commands, e-mail: user-h...@spark.apache.org >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> www.skrasser.com <http://www.skrasser.com/?utm_source=sig> >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> www.skrasser.com <http://www.skrasser.com/?utm_source=sig> >>>> >>>> >>>> >>> >>> >>> -- >>> www.skrasser.com <http://www.skrasser.com/?utm_source=sig> >>> >> >> >
