QFW created SPARK-37997:
---------------------------
Summary: Allow query parameters to be passed into spark.read
Key: SPARK-37997
URL: https://issues.apache.org/jira/browse/SPARK-37997
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 3.2.0
Reporter: QFW
This ticket is to allow query parameters to be used spark.read.
While it is possible to inject some parameters into the query using string
concatenation, this doesn't work for all data types, for example binaries. In
this example, the parameter rowversion is a binary which needs to be passed
into the sql query.
{code:java}
_select_sql = f'SELECT * FROM dbo.Table WHERE RowVersion > {rowversion}'
df = spark.read.format("jdbc") \
.option("url",
"jdbc:sqlserver://databaseserver.database.windows.net;databaseName=databasename")
\
.option("query", _select_sql) \
.option("username", "sql_username") \
.option("password", "sql_password") \
.option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver") \
.load() {code}
This results in the query looking this this...
{code:java}
SELECT * FROM dbo.Address WHERE RowVersion >
bytearray(b'\x00\x00\x00\x00\x02\xdf=\xf5') {code}
As far as I know, there is no way to do this currently.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
