Re: Spark SQL joins taking too long

n(10).registerTempTable(“counties”); > … > And then doing the join. > > > From: Raghu Ganti [mailto:raghuki...@gmail.com] > Sent: Thursday, January 28, 2016 3:06 AM > To: Ted Yu; Дмитро Попович > Cc: user > Subject: Re: Spark SQL joins taking too long > > The pr

Re: Spark SQL joins taking too long

The problem is with the way Spark query plan is being created, IMO, what was happening before is that the order of the tables mattered and when the larger table is given first, it took a very long time (~53mins to complete). I changed the order of the tables with the smaller one first (including

Re: Spark SQL joins taking too long

Yes, the SomeUDF is Contains, shape is a UDT that maps a custom geometry type to sql binary type. Custom geometry type is a Java class. Please let me know if you need further info. Regards Raghu > On Jan 26, 2016, at 17:13, Ted Yu wrote: > > What's the type of shape

Re: Scala MatchError in Spark SQL

ase class, > rather than as a Java class. Scala case classes implement the scala.Product > trait, which Catalyst is looking for. > > > Thanks, > > Andy. > > -- > > Andy Grove > Chief Architect > AgilData - Simple Streaming SQL that Scales > www.agildata.com

Re: Scala MatchError in Spark SQL

Chief Architect > AgilData - Simple Streaming SQL that Scales > www.agildata.com > > > On Wed, Jan 20, 2016 at 10:19 AM, Raghu Ganti <raghuki...@gmail.com> > wrote: > >> Thanks for your reply, Andy. >> >> Yes, that is what I concluded based on the S

Re: Scala MatchError in Spark SQL

Is it not internal to the Catalyst implementation? I should not be modifying the Spark source to get things to work, do I? :-) On Wed, Jan 20, 2016 at 12:21 PM, Raghu Ganti <raghuki...@gmail.com> wrote: > Case classes where? > > On Wed, Jan 20, 2016 at 12:21 PM, Andy

Re: Scala MatchError in Spark SQL

Thanks for your reply, Andy. Yes, that is what I concluded based on the Stack trace. The problem is stemming from Java implementation of generics, but I thought this will go away if you compiled against Java 1.8, which solves the issues of proper generic implementation. Any ideas? Also, are you

Re: Spark SQL create table

This requires Hive to be installed and uses HiveContext, right? What is the SparkSQLContext useful for? On Mon, Jan 18, 2016 at 1:27 PM, Ted Yu wrote: > Please take a look > at sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDataFrameSuite.scala > > On Mon, Jan 18,

Re: Spark SQL create table

Btw, Thanks a lot for all your quick responses - it is very useful and definitely appreciate it :-) On Mon, Jan 18, 2016 at 1:28 PM, Raghu Ganti <raghuki...@gmail.com> wrote: > This requires Hive to be installed and uses HiveContext, right? > > What is the SparkSQLContext useful

Re: Spark SQL create table

Hive, etc > > Regards > > On Mon, Jan 18, 2016 at 10:28 AM, Raghu Ganti <raghuki...@gmail.com> > wrote: > >> This requires Hive to be installed and uses HiveContext, right? >> >> What is the SparkSQLContext useful for? >> >> On Mon, Jan 18, 2016 at

Re: SQL UDF problem (with re to types)

for UDFs defined in Scala, but we can't do > it in Java because the types are erased by the compiler. If you want to > use double you should cast before calling the UDF. > > On Wed, Jan 13, 2016 at 8:10 PM, Raghu Ganti <raghuki...@gmail.com> wrote: > >> So, when I t

Re: SQL UDF problem (with re to types)

So, when I try BigDecimal, it works. But, should it not parse based on what the UDF defines? Am I missing something here? On Wed, Jan 13, 2016 at 4:57 PM, Ted Yu wrote: > Please take a look > at sql/hive/src/test/java/org/apache/spark/sql/hive/aggregate/MyDoubleSum.java >