[
https://issues.apache.org/jira/browse/SPARK-26128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-26128.
-------------------------------
Resolution: Cannot Reproduce
> filter breaks input_file_name
> -----------------------------
>
> Key: SPARK-26128
> URL: https://issues.apache.org/jira/browse/SPARK-26128
> Project: Spark
> Issue Type: Bug
> Components: Spark Shell
> Affects Versions: 2.3.2
> Reporter: Paul Praet
> Priority: Minor
>
> This works:
> {code:java}
> scala>
> spark.read.parquet("/tmp/newparquet").select(input_file_name).show(5,false)
> +-----------------------------------------------------------------------------------------------------------------------------------------------------+
> |input_file_name()
> |
> +-----------------------------------------------------------------------------------------------------------------------------------------------------+
> |file:///tmp/newparquet/parquet-5-PT6H/junit/data/tenant=NA/year=2017/month=201704/day=20170406/hour=2017040618/data.eu-west-1b.290.PT6H.FINAL.parquet|
> |file:///tmp/newparquet/parquet-5-PT6H/junit/data/tenant=NA/year=2017/month=201704/day=20170406/hour=2017040618/data.eu-west-1b.290.PT6H.FINAL.parquet|
> |file:///tmp/newparquet/parquet-5-PT6H/junit/data/tenant=NA/year=2017/month=201704/day=20170406/hour=2017040618/data.eu-west-1b.290.PT6H.FINAL.parquet|
> |file:///tmp/newparquet/parquet-5-PT6H/junit/data/tenant=NA/year=2017/month=201704/day=20170406/hour=2017040618/data.eu-west-1b.290.PT6H.FINAL.parquet|
> |file:///tmp/newparquet/parquet-5-PT6H/junit/data/tenant=NA/year=2017/month=201704/day=20170406/hour=2017040618/data.eu-west-1b.290.PT6H.FINAL.parquet|
> +-----------------------------------------------------------------------------------------------------------------------------------------------------+
> {code}
> When adding a filter:
> {code:java}
> scala>
> spark.read.parquet("/tmp/newparquet").where("key.station='XYZ'").select(input_file_name()).show(5,false)
> +-----------------+
> |input_file_name()|
> +-----------------+
> | |
> | |
> | |
> | |
> | |
> +-----------------+
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
