[
https://issues.apache.org/jira/browse/PHOENIX-6667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Istvan Toth resolved PHOENIX-6667.
----------------------------------
Fix Version/s: connectors-6.0.0
Resolution: Fixed
Committed.
Thanks a lot for finding the solution [~attilapiros] , and for the review
[~richardantal] .
> Spark3 connector requires that all columns are specified when writing
> ---------------------------------------------------------------------
>
> Key: PHOENIX-6667
> URL: https://issues.apache.org/jira/browse/PHOENIX-6667
> Project: Phoenix
> Issue Type: Bug
> Components: connectors, spark-connector
> Affects Versions: connectors-6.0.0
> Reporter: Istvan Toth
> Assignee: Attila Zsolt Piros
> Priority: Major
> Fix For: connectors-6.0.0
>
>
> For Spark 2, it was possible to omit some columns from the dataframe, the
> same way it is not mandatory to specify all columns when upserting via SQL.
> Spark3 has added new checks, which require that EVERY sql column is specifed
> in the DataFrame.
> Consequently, when using the current API, writing will fail unless you
> specify all columns.
> This is a loss of functionality WRT Phoenix (and other SQL datastores)
> compared to Spark2.
> I don't think that we can do anything from the Phoenix side, just documenting
> the regression here.
> Maybe future Spark versions will make this configurable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
