[
https://issues.apache.org/jira/browse/HUDI-1745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-1745:
--
Description:
[https://github.com/apache/hudi/issues/2748]
PR thats of interest: [https://github.com/apache/hudi/pull/2431]
I see we have three options. Let me know if we have more.
Option1:
Similar to SparkRowSerDe, we might have to introduce an interface for
translateSqlOptions and override based on spark versions. But already we have
two sub modules for spark2 and spark3. and now we might have to add more such
modules for diff spark2 versions which might need more thought to do it
elegantly.
Option2:
Since this feature is added only w/ 0.8.0, and not like a more sought after
feature, we could revert this commit and unblock ourselves for 0.8.0. Once
release is complete, we can decide how to do about doing this and get this
feature in for next release.
Option3:
we say that hudi does not support spark version < 2.4.4 w/ 0.8.0. Don't think
we can go this route. But just listing it out.
was:
[https://github.com/apache/hudi/issues/2748]
PR thats of interest: https://github.com/apache/hudi/pull/2431
> Hudi compilation fails w/ spark version < 2.4.4 due to usage of unavailable
> spark api
> -
>
> Key: HUDI-1745
> URL: https://issues.apache.org/jira/browse/HUDI-1745
> Project: Apache Hudi
> Issue Type: Bug
> Components: Spark Integration
>Affects Versions: 0.8.0
>Reporter: sivabalan narayanan
>Priority: Major
> Labels: sev:critical
>
> [https://github.com/apache/hudi/issues/2748]
>
> PR thats of interest: [https://github.com/apache/hudi/pull/2431]
>
> I see we have three options. Let me know if we have more.
> Option1:
> Similar to SparkRowSerDe, we might have to introduce an interface for
> translateSqlOptions and override based on spark versions. But already we have
> two sub modules for spark2 and spark3. and now we might have to add more such
> modules for diff spark2 versions which might need more thought to do it
> elegantly.
> Option2:
> Since this feature is added only w/ 0.8.0, and not like a more sought after
> feature, we could revert this commit and unblock ourselves for 0.8.0. Once
> release is complete, we can decide how to do about doing this and get this
> feature in for next release.
> Option3:
> we say that hudi does not support spark version < 2.4.4 w/ 0.8.0. Don't think
> we can go this route. But just listing it out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
