In regards to the large GC pauses, assuming you allocated all 100GB of
memory per worker you may consider running with less memory on your Worker
nodes, or splitting up the available memory on the Worker nodes amongst
several worker instances. The JVM's garbage collection starts to become
very slow as the memory allocation for the heap becomes large. At 100GB it
may not be unusual to see GC take minutes at time. I believe with Yarn or
Standalone clusters you should be able to run multiple smaller JVM
instances on your workers so you can still use your cluster resources but
also won't have a single JVM allocating an unwieldy amount of much memory.
On Mon, May 4, 2015 at 2:23 AM, Nick Travers <n.e.trav...@gmail.com> wrote:
> Could you be more specific in how this is done?
>
> A DataFrame class doesn't have that method.
>
> On Sun, May 3, 2015 at 11:07 PM, ayan guha <guha.a...@gmail.com> wrote:
>
>> You can use custom partitioner to redistribution using partitionby
>> On 4 May 2015 15:37, "Nick Travers" <n.e.trav...@gmail.com> wrote:
>>
>>> I'm currently trying to join two large tables (order 1B rows each) using
>>> Spark SQL (1.3.0) and am running into long GC pauses which bring the job
>>> to
>>> a halt.
>>>
>>> I'm reading in both tables using a HiveContext with the underlying files
>>> stored as Parquet Files. I'm using something along the lines of
>>> HiveContext.sql("SELECT a.col1, b.col2 FROM a JOIN b ON a.col1 =
>>> b.col1") to
>>> set up the join.
>>>
>>> When I execute this (with an action such as .count) I see the first few
>>> stages complete, but the job eventually stalls. The GC counts keep
>>> increasing for each executor.
>>>
>>> Running with 6 workers, each with 2T disk and 100GB RAM.
>>>
>>> Has anyone else run into this issue? I'm thinking I might be running into
>>> issues with the shuffling of the data, but I'm unsure of how to get
>>> around
>>> this? Is there a way to redistribute the rows based on the join key
>>> first,
>>> and then do the join?
>>>
>>> Thanks in advance.
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/Long-GC-pauses-with-Spark-SQL-1-3-0-and-billion-row-tables-tp22750.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>> For additional commands, e-mail: user-h...@spark.apache.org
>>>
>>>
>
