Impala created parquet tables

2015-06-20 Thread Debasish Das
Hi, I have some impala created parquet tables which hive 0.13.2 can read fine. Now the same table when I want to read using Spark SQL 1.3 I am getting exception class exception that parquet.hive.serde.ParquetHiveSerde not found. I am assuming that hive somewhere is putting the

Re: [Tungsten] NPE in UnsafeShuffleWriter.java

2015-06-20 Thread Josh Rosen
I've filed https://issues.apache.org/jira/browse/SPARK-8498 to fix this error-handling code. On Fri, Jun 19, 2015 at 11:51 AM, Josh Rosen rosenvi...@gmail.com wrote: Hey Peter, I think that this is actually due to an error-handling issue: if you look at the stack trace that you posted, the

Velox Model Server

2015-06-20 Thread Debasish Das
Hi, The demo of end-to-end ML pipeline including the model server component at Spark Summit was really cool. I was wondering if the Model Server component is based upon Velox or it uses a completely different architecture. https://github.com/amplab/velox-modelserver We are looking for an open

Re: Welcoming some new committers

2015-06-20 Thread Debasish Das
Congratulations to All. DB great work in bringing quasi newton methods to Spark ! On Wed, Jun 17, 2015 at 3:18 PM, Chester Chen ches...@alpinenow.com wrote: Congratulations to All. DB and Sandy, great works ! On Wed, Jun 17, 2015 at 3:12 PM, Matei Zaharia matei.zaha...@gmail.com wrote:

Fwd: Verifying number of workers in Spark Streaming

2015-06-20 Thread anshu shukla
Any suggestions please ..!! How to know that In stream Processing over the cluster of 8 machines all the machines/woker nodes are being used (my cluster have 8 slaves ) . I am submitting job from master itself over the ec-2 cluster crated by the ec-2 scripts available with spark. But i am

Re: Welcoming some new committers

2015-06-20 Thread Andrew Or
Welcome! 2015-06-20 7:30 GMT-07:00 Debasish Das debasish.da...@gmail.com: Congratulations to All. DB great work in bringing quasi newton methods to Spark ! On Wed, Jun 17, 2015 at 3:18 PM, Chester Chen ches...@alpinenow.com wrote: Congratulations to All. DB and Sandy, great works !

[pyspark][mllib] What is the best way to treat int and long int between python2.6/python3.4 and Java?

2015-06-20 Thread Yu Ishikawa
Hi all, I have done a survey about treating int type and long int type between python2.6/python3.4 and Java as follows. When we want to return Long value(s) from Java to python vice versa, what is the best way? https://gist.github.com/yu-iskw/12e92c2d718ca41dea90 Based on that, Joseph and I are

Re: Welcoming some new committers

2015-06-20 Thread Ted Malaska
Super congrats. Well earned. On Jun 20, 2015 12:48 PM, Andrew Or and...@databricks.com wrote: Welcome! 2015-06-20 7:30 GMT-07:00 Debasish Das debasish.da...@gmail.com: Congratulations to All. DB great work in bringing quasi newton methods to Spark ! On Wed, Jun 17, 2015 at 3:18 PM,

Re: Welcoming some new committers

2015-06-20 Thread Guru Medasani
Congratulations to all the new committers! Guru Medasani gdm...@gmail.com On Jun 17, 2015, at 5:12 PM, Matei Zaharia matei.zaha...@gmail.com wrote: Hey all, Over the past 1.5 months we added a number of new committers to the project, and I wanted to welcome them now that all of their