Re: SparkSQL with large result size

[Buntu Dev]

Thanks Chris for pointing out the issue. I think I was able to get over
this issue by:

- repartitioning to increase the number of partitions (about 6k partitions)
- apply sort() on the resulting dataframe to coalesce into single sorted
partition file
- read the sorted file and then adding just limit() to get the desired
number of rows seem to have worked

Thanks everyone for the input!

On Tue, May 10, 2016 at 1:20 AM, Christophe Préaud <
christophe.pre...@kelkoo.com> wrote:

> Hi,
>
> You may be hitting this bug: SPARK-9879
> 
>
> In other words: did you try without the LIMIT clause?
>
> Regards,
> Christophe.
>
>
> On 02/05/16 20:02, Gourav Sengupta wrote:
>
> Hi,
>
> I have worked on 300GB data by querying it  from CSV (using SPARK CSV)
>  and writing it to Parquet format and then querying parquet format to query
> it and partition the data and write out individual csv files without any
> issues on a single node SPARK cluster installation.
>
> Are you trying to cache in the entire data? What is that you are trying to
> achieve in your used case?
>
> Regards,
> Gourav
>
> On Mon, May 2, 2016 at 5:59 PM, Ted Yu  wrote:
>
>> That's my interpretation.
>>
>> On Mon, May 2, 2016 at 9:45 AM, Buntu Dev < 
>> buntu...@gmail.com> wrote:
>>
>>> Thanks Ted, I thought the avg. block size was already low and less than
>>> the usual 128mb. If I need to reduce it further via parquet.block.size, it
>>> would mean an increase in the number of blocks and that should increase the
>>> number of tasks/executors. Is that the correct way to interpret this?
>>>
>>> On Mon, May 2, 2016 at 6:21 AM, Ted Yu < 
>>> yuzhih...@gmail.com> wrote:
>>>
 Please consider decreasing block size.

 Thanks

 > On May 1, 2016, at 9:19 PM, Buntu Dev < 
 buntu...@gmail.com> wrote:
 >
 > I got a 10g limitation on the executors and operating on parquet
 dataset with block size 70M with 200 blocks. I keep hitting the memory
 limits when doing a 'select * from t1 order by c1 limit 100' (ie, 1M).
 It works if I limit to say 100k. What are the options to save a large
 dataset without running into memory issues?
 >
 > Thanks!

>>>
>>>
>>
>
>
> --
> Kelkoo SAS
> Société par Actions Simplifiée
> Au capital de € 4.168.964,30
> Siège social : 158 Ter Rue du Temple 75003 Paris
> 425 093 069 RCS Paris
>
> Ce message et les pièces jointes sont confidentiels et établis à
> l'attention exclusive de leurs destinataires. Si vous n'êtes pas le
> destinataire de ce message, merci de le détruire et d'en avertir
> l'expéditeur.
>

Re: SparkSQL with large result size

[Christophe Préaud]

Hi,

You may be hitting this bug: 
SPARK-9879

In other words: did you try without the LIMIT clause?

Regards,
Christophe.

On 02/05/16 20:02, Gourav Sengupta wrote:
Hi,

I have worked on 300GB data by querying it  from CSV (using SPARK CSV)  and 
writing it to Parquet format and then querying parquet format to query it and 
partition the data and write out individual csv files without any issues on a 
single node SPARK cluster installation.

Are you trying to cache in the entire data? What is that you are trying to 
achieve in your used case?

Regards,
Gourav

On Mon, May 2, 2016 at 5:59 PM, Ted Yu 
mailto:yuzhih...@gmail.com>> wrote:
That's my interpretation.

On Mon, May 2, 2016 at 9:45 AM, Buntu Dev 
<buntu...@gmail.com> 
wrote:
Thanks Ted, I thought the avg. block size was already low and less than the 
usual 128mb. If I need to reduce it further via parquet.block.size, it would 
mean an increase in the number of blocks and that should increase the number of 
tasks/executors. Is that the correct way to interpret this?

On Mon, May 2, 2016 at 6:21 AM, Ted Yu 
<yuzhih...@gmail.com> 
wrote:
Please consider decreasing block size.

Thanks

> On May 1, 2016, at 9:19 PM, Buntu Dev 
> <buntu...@gmail.com> 
> wrote:
>
> I got a 10g limitation on the executors and operating on parquet dataset with 
> block size 70M with 200 blocks. I keep hitting the memory limits when doing a 
> 'select * from t1 order by c1 limit 100' (ie, 1M). It works if I limit to 
> say 100k. What are the options to save a large dataset without running into 
> memory issues?
>
> Thanks!






Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 158 Ter Rue du Temple 75003 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à l'attention 
exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce 
message, merci de le détruire et d'en avertir l'expéditeur.

Re: SparkSQL with large result size

[Gourav Sengupta]

Hi,

I have worked on 300GB data by querying it  from CSV (using SPARK CSV)  and
writing it to Parquet format and then querying parquet format to query it
and partition the data and write out individual csv files without any
issues on a single node SPARK cluster installation.

Are you trying to cache in the entire data? What is that you are trying to
achieve in your used case?

Regards,
Gourav

On Mon, May 2, 2016 at 5:59 PM, Ted Yu  wrote:

> That's my interpretation.
>
> On Mon, May 2, 2016 at 9:45 AM, Buntu Dev  wrote:
>
>> Thanks Ted, I thought the avg. block size was already low and less than
>> the usual 128mb. If I need to reduce it further via parquet.block.size, it
>> would mean an increase in the number of blocks and that should increase the
>> number of tasks/executors. Is that the correct way to interpret this?
>>
>> On Mon, May 2, 2016 at 6:21 AM, Ted Yu  wrote:
>>
>>> Please consider decreasing block size.
>>>
>>> Thanks
>>>
>>> > On May 1, 2016, at 9:19 PM, Buntu Dev  wrote:
>>> >
>>> > I got a 10g limitation on the executors and operating on parquet
>>> dataset with block size 70M with 200 blocks. I keep hitting the memory
>>> limits when doing a 'select * from t1 order by c1 limit 100' (ie, 1M).
>>> It works if I limit to say 100k. What are the options to save a large
>>> dataset without running into memory issues?
>>> >
>>> > Thanks!
>>>
>>
>>
>

Re: SparkSQL with large result size

[Ted Yu]

That's my interpretation.

On Mon, May 2, 2016 at 9:45 AM, Buntu Dev  wrote:

> Thanks Ted, I thought the avg. block size was already low and less than
> the usual 128mb. If I need to reduce it further via parquet.block.size, it
> would mean an increase in the number of blocks and that should increase the
> number of tasks/executors. Is that the correct way to interpret this?
>
> On Mon, May 2, 2016 at 6:21 AM, Ted Yu  wrote:
>
>> Please consider decreasing block size.
>>
>> Thanks
>>
>> > On May 1, 2016, at 9:19 PM, Buntu Dev  wrote:
>> >
>> > I got a 10g limitation on the executors and operating on parquet
>> dataset with block size 70M with 200 blocks. I keep hitting the memory
>> limits when doing a 'select * from t1 order by c1 limit 100' (ie, 1M).
>> It works if I limit to say 100k. What are the options to save a large
>> dataset without running into memory issues?
>> >
>> > Thanks!
>>
>
>

Re: SparkSQL with large result size

[Buntu Dev]

Thanks Ted, I thought the avg. block size was already low and less than the
usual 128mb. If I need to reduce it further via parquet.block.size, it
would mean an increase in the number of blocks and that should increase the
number of tasks/executors. Is that the correct way to interpret this?

On Mon, May 2, 2016 at 6:21 AM, Ted Yu  wrote:

> Please consider decreasing block size.
>
> Thanks
>
> > On May 1, 2016, at 9:19 PM, Buntu Dev  wrote:
> >
> > I got a 10g limitation on the executors and operating on parquet dataset
> with block size 70M with 200 blocks. I keep hitting the memory limits when
> doing a 'select * from t1 order by c1 limit 100' (ie, 1M). It works if
> I limit to say 100k. What are the options to save a large dataset without
> running into memory issues?
> >
> > Thanks!
>

Re: SparkSQL with large result size

[Ted Yu]

Please consider decreasing block size. 

Thanks

> On May 1, 2016, at 9:19 PM, Buntu Dev  wrote:
> 
> I got a 10g limitation on the executors and operating on parquet dataset with 
> block size 70M with 200 blocks. I keep hitting the memory limits when doing a 
> 'select * from t1 order by c1 limit 100' (ie, 1M). It works if I limit to 
> say 100k. What are the options to save a large dataset without running into 
> memory issues?
> 
> Thanks!

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: SparkSQL with large result size

[ayan guha]

How many executors are you running? Is your partition scheme ensures data
is distributed evenly? It is possible that your data is skewed and one of
the executors failing. Maybe you can try reduce per executor memory and
increase partitions.
On 2 May 2016 14:19, "Buntu Dev"  wrote:

> I got a 10g limitation on the executors and operating on parquet dataset
> with block size 70M with 200 blocks. I keep hitting the memory limits when
> doing a 'select * from t1 order by c1 limit 100' (ie, 1M). It works if
> I limit to say 100k. What are the options to save a large dataset without
> running into memory issues?
>
> Thanks!
>