[
https://issues.apache.org/jira/browse/SPARK-31836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117726#comment-17117726
]
Wesley Hildebrandt commented on SPARK-31836:
--------------------------------------------
One last note .. with only four (or fewer) files this doesn't happen, I suspect
because PySpark uses four executors so each only takes a single input file.
> input_file_name() gives wrong value following Python UDF usage
> --------------------------------------------------------------
>
> Key: SPARK-31836
> URL: https://issues.apache.org/jira/browse/SPARK-31836
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Wesley Hildebrandt
> Priority: Major
>
> I'm using PySpark for Spark 3.0.0 RC1 with Python 3.6.8.
> The following commands demonstrate that the input_file_name() function
> sometimes returns the wrong filename following usage of a Python UDF:
> $ for i in `seq 5`; do echo $i > /tmp/test-file-$i; done
> $ pyspark
> >>> import pyspark.sql.functions as F
> >>> spark.readStream.text('file:///tmp/test-file-*',
> >>> wholetext=True).withColumn('file1',
> >>> F.input_file_name()).withColumn('udf', F.udf(lambda
> >>> x:x)('value')).withColumn('file2',
> >>> F.input_file_name()).writeStream.trigger(once=True).foreachBatch(lambda
> >>> df,_: df.select('file1','file2').show(truncate=False,
> >>> vertical=True)).start().awaitTermination()
> A few notes about this bug:
> * It happens with many different files, so it's not related to the file
> contents
> * It also happens loading files from HDFS, so storage location is not a
> factor
> * It also happens using .csv() to read the files instead of .text(), so
> input format is not a factor
> * I have not been able to cause the error without using readStream, so it
> seems to be related to streaming
> * The bug also happens using spark-submit to send a job to my cluster
> * I haven't tested an older version, but it's possible that Spark pulls
> 24958 and 25321([https://github.com/apache/spark/pull/24958],
> [https://github.com/apache/spark/pull/25321]) to fix issue 28153
> (https://issues.apache.org/jira/browse/SPARK-28153) introduced this bug?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
