[
https://issues.apache.org/jira/browse/SPARK-11522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993212#comment-14993212
]
Xin Wu commented on SPARK-11522:
--------------------------------
Actually this seems not related to whether the table is created as external
table or not.. it seems to be related to table file format. When the table is
created stored as parquet, regardless of external or not,
{code}
scala> sqlContext.sql("create table parquet_ab (c1 int, c2 int) stored as
parquet as select 1, 2")
scala> sqlContext.sql("select input_file_name() as file from parquet_ab").show
+--------------------+
| file|
+--------------------+
|file:/user/hive/w...|
+--------------------+
scala> sqlContext.sql("create external table parquet_a(c1 int, c2 int) stored
as parquet location 'file:///user/hive/warehouse/parquet_a'")
scala> sqlContext.sql("select input_file_name() as file from parquet_a").show
+--------------------+
| file|
+--------------------+
|file:/user/hive/w...|
+--------------------+
{code}
Non-external, non-parquet table
{code}
scala> sqlContext.sql("CREATE TABLE internal_test(c1 INT, c2 INT) as select 1,
2")
scala> sqlContext.sql("select input_file_name() as file from
internal_test").show
+----+
|file|
+----+
| |
+----+
{code}
> input_file_name() returns "" for external tables
> ------------------------------------------------
>
> Key: SPARK-11522
> URL: https://issues.apache.org/jira/browse/SPARK-11522
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.5.1
> Reporter: Simeon Simeonov
> Labels: external-tables, hive, sql
>
> Given an external table definition where the data consists of many CSV files,
> {{input_file_path()}} returns empty strings.
> Table definition:
> {code}
> CREATE EXTERNAL TABLE external_test(page_id INT, impressions INT)
> ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
> WITH SERDEPROPERTIES (
> "separatorChar" = ",",
> "quoteChar" = "\"",
> "escapeChar" = "\\"
> )
> LOCATION 'file:///Users/sim/spark/test/external_test'
> {code}
> Query:
> {code}
> sql("SELECT input_file_name() as file FROM external_test").show
> {code}
> Output:
> {code}
> +----+
> |file|
> +----+
> | |
> | |
> ...
> | |
> +----+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
