Spark Powered By Page

2015-11-16 Thread Alex Rovner
I would like to list our organization on the Powered by Page. Company: Magnetic Description: We are leveraging Spark Core, Streaming and YARN to process our massive datasets. *Alex Rovner* *Director, Data Engineering * *o:* 646.759.0052 * <http://www.magnetic.com/>*

YARN Labels

2015-11-16 Thread Alex Rovner
ion master not to run on spot nodes. For what ever reason, application master is not able to recover in cases the node where it was running suddenly disappears, which is the case with spot nodes. Any guidance on this topic is appreciated. *Alex Rovner* *Director, Data Engineering * *o:* 646.759.005

Re: Why dataframe.persist(StorageLevels.MEMORY_AND_DISK_SER) hangs for long time?

2015-10-10 Thread Alex Rovner
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org <javascript:;> > For additional commands, e-mail: user-h...@spark.apache.org <javascript:;> > > -- *Alex Rovner* *Director, Data Engineering * *o:* 646.759.0052 * <http://www.magnetic.com/>*

Re: [Spark on YARN] Multiple Auxiliary Shuffle Service Versions

2015-10-06 Thread Alex Rovner
Thank you all for your help. *Alex Rovner* *Director, Data Engineering * *o:* 646.759.0052 * <http://www.magnetic.com/>* On Tue, Oct 6, 2015 at 11:17 AM, Steve Loughran <ste...@hortonworks.com> wrote: > > On 6 Oct 2015, at 01:23, Andrew Or <and...@databricks.com> wro

Re: [Spark on YARN] Multiple Auxiliary Shuffle Service Versions

2015-10-05 Thread Alex Rovner
to configure multiple versions to use the same shuffling service. *Alex Rovner* *Director, Data Engineering * *o:* 646.759.0052 * <http://www.magnetic.com/>* On Mon, Oct 5, 2015 at 11:06 AM, Andreas Fritzler < andreas.fritz...@gmail.com> wrote: > Hi Steve, Alex, > &

Re: [Spark on YARN] Multiple Auxiliary Shuffle Service Versions

2015-10-05 Thread Alex Rovner
I have the same question about the history server. We are trying to run multiple versions of Spark and are wondering if the history server is backwards compatible. *Alex Rovner* *Director, Data Engineering * *o:* 646.759.0052 * <http://www.magnetic.com/>* On Mon, Oct 5, 2015 at 9:22 AM, A

Re: [Spark on YARN] Multiple Auxiliary Shuffle Service Versions

2015-10-05 Thread Alex Rovner
Hey Steve, Are you referring to the 1.5 version of the history server? *Alex Rovner* *Director, Data Engineering * *o:* 646.759.0052 * <http://www.magnetic.com/>* On Mon, Oct 5, 2015 at 10:18 AM, Steve Loughran <ste...@hortonworks.com> wrote: > > > On 5 Oct 2015,

Re: How to optimize group by query fired using hiveContext.sql?

2015-10-04 Thread Alex Rovner
Can you at least copy paste the error(s) you are seeing when the job fails? Without the error message(s), it's hard to even suggest anything. *Alex Rovner* *Director, Data Engineering * *o:* 646.759.0052 * <http://www.magnetic.com/>* On Sat, Oct 3, 2015 at 9:50 AM, Umesh Kacha <

Re: How to optimize group by query fired using hiveContext.sql?

2015-10-03 Thread Alex Rovner
he command line you are using to submit your jobs for further troubleshooting. *Alex Rovner* *Director, Data Engineering * *o:* 646.759.0052 * <http://www.magnetic.com/>* On Sat, Oct 3, 2015 at 6:19 AM, unk1102 <umesh.ka...@gmail.com> wrote: > Hi I have couple of Spark jobs which

Re: How to optimize group by query fired using hiveContext.sql?

2015-10-03 Thread Alex Rovner
Can you send over your yarn logs along with the command you are using to submit your job? *Alex Rovner* *Director, Data Engineering * *o:* 646.759.0052 * <http://www.magnetic.com/>* On Sat, Oct 3, 2015 at 9:07 AM, Umesh Kacha <umesh.ka...@gmail.com> wrote: > Hi Alex thanks muc