Wesley Hildebrandt created SPARK-31836: ------------------------------------------
Summary: input_file_name() gives wrong value following Python UDF usage Key: SPARK-31836 URL: https://issues.apache.org/jira/browse/SPARK-31836 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 3.0.0 Reporter: Wesley Hildebrandt I'm using PySpark for Spark 3.0.0 RC1 with Python 3.6.8. The following commands demonstrate that the input_file_name() function sometimes returns the wrong filename following usage of a Python UDF: $ for i in `seq 5`; do echo $i > /tmp/test-file-$i; done $ pyspark >>> import pyspark.sql.functions as F >>> spark.readStream.text('file:///tmp/test-file-*', >>> wholetext=True).withColumn('file1', F.input_file_name()).withColumn('udf', >>> F.udf(lambda x:x)('value')).withColumn('file2', >>> F.input_file_name()).writeStream.trigger(once=True).foreachBatch(lambda >>> df,_: df.select('file1','file2').show(truncate=False, >>> vertical=True)).start().awaitTermination() A few notes about this bug: * It happens with many different files, so it's not related to the file contents * It also happens loading files from HDFS, so storage location is not a factor * It also happens using .csv() to read the files instead of .text(), so input format is not a factor * I have not been able to cause the error without using readStream, so it seems to be related to streaming * The bug also happens using spark-submit to send a job to my cluster * I haven't tested an older version, but it's possible that Spark pulls 24958 and 25321([https://github.com/apache/spark/pull/24958], [https://github.com/apache/spark/pull/25321]) to fix issue 28153 (https://issues.apache.org/jira/browse/SPARK-28153) introduced this bug? -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org