Hi,
I guess this is fixed by https://github.com/apache/spark/pull/3110
which is not for complex type casting but makes inserting into hive
table be able to handle complex types ignoring nullability.
I also sent a pull-request (https://github.com/apache/spark/pull/3150)
for complex type casting
Thanks Takuya! Will take a look into it later. And sorry for not being
able to review all the PRs in time recently (mostly because of rushing
Spark 1.2 release and Thanksgiving :) ).
On 11/27/14 1:35 AM, Takuya UESHIN wrote:
Hi,
I guess this is fixed by
https://github.com/apache/spark/blob/84d79ee9ec47465269f7b0a7971176da93c96f3f/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
Doesn't look like spark sql support nested complex types right now
--
View this message in context:
Exactly that seems to be the problem will have to wait for the next release
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-insert-complex-types-like-map-string-map-string-int-in-spark-sql-tp19603p19734.html
Sent from the Apache Spark User List
Spark SQL supports complex types, but casting doesn't work for complex
types right now.
On 11/25/14 4:04 PM, critikaled wrote:
https://github.com/apache/spark/blob/84d79ee9ec47465269f7b0a7971176da93c96f3f/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
Doesn't
Can you give the full stack trace. You might be hitting:
https://issues.apache.org/jira/browse/SPARK-4293
On Sun, Nov 23, 2014 at 3:00 PM, critikaled isasmani@gmail.com wrote:
Hi,
I am trying to insert particular set of data from rdd to a hive table I
have Map[String,Map[String,Int]] in
Thanks for the reply Micheal here is the stack trace
org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in
stage 0.0 failed 1 times, most recent failure: Lost task 3.0 in stage 0.0
(TID 3, localhost): scala.MatchError: MapType(StringType,StringType,true)
(of class
Hi,
I am trying to insert particular set of data from rdd to a hive table I
have Map[String,Map[String,Int]] in scala which I want to insert into the
table of mapstring,maplt;string,int I was able to create the table but
while inserting it says scala.MatchError:
