[Problem Solved]Re: Spark partition size tuning

2016-01-27 Thread Jia Zou
need to tune Spark data partition size for input data >>> that is stored in Tachyon (default is 512MB), but above method can't work >>> for Tachyon data. >>> >>> Do you have any suggestions? Thanks very much! >>> >>> Best Regards, >>> Jia >&g

Re: Spark partition size tuning

2016-01-27 Thread Jia Zou
n't work >> for Tachyon data. >> >> Do you have any suggestions? Thanks very much! >> >> Best Regards, >> Jia >> >> >> -- Forwarded message -- >> From: Jia Zou <jacqueline...@gmail.com> >> Date: Thu, Jan 21, 2016 at 10:05 PM >>

Re: Spark partition size tuning

2016-01-26 Thread Gene Pang
- > From: Jia Zou <jacqueline...@gmail.com> > Date: Thu, Jan 21, 2016 at 10:05 PM > Subject: Spark partition size tuning > To: "user @spark" <user@spark.apache.org> > > > Dear all! > > When using Spark to read from local file system, the default

Re: Spark partition size tuning

2016-01-26 Thread Pavel Plotnikov
any suggestions? Thanks very much! > > Best Regards, > Jia > > > -- Forwarded message -- > From: Jia Zou <jacqueline...@gmail.com> > Date: Thu, Jan 21, 2016 at 10:05 PM > Subject: Spark partition size tuning > To: "user @spark" <user@spark.a

Fwd: Spark partition size tuning

2016-01-25 Thread Jia Zou
thod can't work for Tachyon data. Do you have any suggestions? Thanks very much! Best Regards, Jia -- Forwarded message -- From: Jia Zou <jacqueline...@gmail.com> Date: Thu, Jan 21, 2016 at 10:05 PM Subject: Spark partition size tuning To: "user @spark" <user

Spark partition size tuning

2016-01-21 Thread Jia Zou
Dear all! When using Spark to read from local file system, the default partition size is 32MB, how can I increase the partition size to 128MB, to reduce the number of tasks? Thank you very much! Best Regards, Jia