Alex Ott created SPARK-55087:
--------------------------------
Summary: Reading with Python Data source fails by default if it
implements pushFilters
Key: SPARK-55087
URL: https://issues.apache.org/jira/browse/SPARK-55087
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.1
Reporter: Alex Ott
There is a usability issue in custom Python Data sources. I'm implementing
pushFilters for my source, but when I'm trying to use it, it fails with the
error:
pyspark.errors.exceptions.base.PySparkAssertionError:
[DATA_SOURCE_PUSHDOWN_DISABLED] CassandraBatchReader implements pushFilters()
but filter pushdown is disabled because configuration
'spark.sql.python.filterPushdown.enabled' is false. Set it to true to enable
filter pushdown.
If this functionality is disabled by default, why not simply skip calling the
pushFilters?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
