[
https://issues.apache.org/jira/browse/HUDI-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sreeram Ramji updated HUDI-1153:
--------------------------------
Status: In Progress (was: Open)
> Spark DataSource and Streaming Write must fail when operation type is
> misconfigured
> -----------------------------------------------------------------------------------
>
> Key: HUDI-1153
> URL: https://issues.apache.org/jira/browse/HUDI-1153
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Spark Integration
> Reporter: Balaji Varadarajan
> Assignee: Sreeram Ramji
> Priority: Major
> Fix For: 0.6.1
>
>
> Context: [https://github.com/apache/hudi/issues/1902#issuecomment-669698259]
>
> If you look at DataSourceUtils.java,
> [https://github.com/apache/hudi/blob/master/hudi-spark/src/main/java/org/apache/hudi/DataSourceUtils.java#L257]
>
> we are string comparison to determine operation type which is a bad idea and
> a typo could result in "upsert" being used silently.
>
> Just like
> [https://github.com/apache/hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java#L187]
> being used for DeltaStreamer, we need similar enums defined in
> DataSourceOptions.scala for OPERATION_OPT_KEY but care must be taken to
> ensure we do not cause backwards compatibility issue by changing the property
> value. In other words, we need to retain the lower case values
> ("bulk_insert", "insert" and "upsert") but make it an enum.
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
