Hi Yana,
I was using sqlContext in the program by creating new SqlContext(sc). This was
created the problem when i submit the job using spark-submit. Where as, when I
run the same program in spark-shell, the default context is hive context (it
seems ) and every thing seems to be fine. This created confusion.
As solution, i called new HiveContext(sc) instead of SqlContext.
cheerskiran.
On Wednesday, May 20, 2015 6:38 PM, yana <yana.kadiy...@gmail.com> wrote:
Just a guess but are you using HiveContext in one case vs SqlContext
inanother? You dont show a stacktrace but this looks like parser error...Which
would make me guess different context or different spark versio on the cluster
you are submitting to...
Sent on the new Sprint Network from my Samsung Galaxy S®4.
-------- Original message --------From: kiran mavatoor Date:05/20/2015 5:57 AM
(GMT-05:00) To: User Subject: LATERAL VIEW explode issue
Hi,
When I use "LATERAL VIEW explode" on the registered temp table in spark shell,
it works. But when I use the same in spark-submit (as jar file) it is not
working. its giving error - "failure: ``union'' expected but identifier VIEW
found"
sql statement i am using is
SELECT id,mapKey FROM locations LATERAL VIEW
explode(map_keys(jsonStringToMapUdf(countries))) countries AS mapKey
I registered "jsonStringToMapUdf" as my sql function.
ThanksKiran9008099770
