zenglinxi created SPARK-16253:
---------------------------------
Summary: make spark sql compatible with hive sql that using python
script transform like using 'xxx.py'
Key: SPARK-16253
URL: https://issues.apache.org/jira/browse/SPARK-16253
Project: Spark
Issue Type: Task
Components: SQL
Affects Versions: 1.6.2
Reporter: zenglinxi
Some hive sql like:
{quote}
add file /tmp/spark_sql_test/test.py;
select transform(cityname) using 'test.py' as (new_cityname) from
test.spark2_orc where dt='20160622' limit 5 ;
{quote}
can't be executed by spark sql directly, since it will return error like:
{quote}
16/06/26 11:01:28 INFO codegen.GenerateUnsafeProjection: Code generated in
19.054534 ms
16/06/26 11:01:28 ERROR execution.ScriptTransformationWriterThread: /bin/bash:
test.py: command not found
{quote}
and the sql works fine in hive with MR.
Lots of ETL can't be moved from hive to spark sql because of this problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
