Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22104#discussion_r210786895
--- Diff: python/pyspark/sql/tests.py ---
@@ -3367,6 +3367,33 @@ def test_ignore_column_of_all_nulls(self):
finally:
shutil.rmtree(path)
+ # SPARK-24721
+ def test_datasource_with_udf_filter_lit_input(self):
+ from pyspark.sql.functions import udf, lit, col
+
+ path = tempfile.mkdtemp()
+ shutil.rmtree(path)
+ try:
+
self.spark.range(1).write.mode("overwrite").format('csv').save(path)
+ filesource_df = self.spark.read.csv(path)
+ datasource_df = self.spark.read \
+ .format("org.apache.spark.sql.sources.SimpleScanSource") \
+ .option('from', 0).option('to', 1).load()
+ datasource_v2_df = self.spark.read \
+
.format("org.apache.spark.sql.sources.v2.SimpleDataSourceV2") \
--- End diff --
This wouldn't work if test classes are not compiled. I think we should
better make another test suite that skips the test if the test classes are not
existent.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
