[
https://issues.apache.org/jira/browse/HUDI-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ying Lin updated HUDI-6221:
---
Description:
When using flink 1.13.6 and hudi 0.13.0 cow + append + clustering mode, if the
field list contains map type and aysnc clustering job scheduled, will throw
exception:
{quote}The requested schema is not compatible with the file schema.
incompatible types: required binary key (STRING) != optional binary key (STRING)
{quote}
Root reason is [HUDI-3378|https://github.com/apache/hudi/pull/7345] change
parquet reader. The latest parquet reader is compatible with spark but not
fully compatible with flink due to flink parquet schema is different from spark
parquet schema.
We will make two patch, the first patch fix this bug in 0.13.x. The last patch
fix diff schema between flink parquet and spark parquet.
was:
When using flink 1.13.6 and hudi 0.13.0 cow + append + clustering mode, if the
field list contains map type and aysnc clustering job scheduled, will throw
exception:
The requested schema is not compatible with the file schema. incompatible
types: required binary key (STRING) != optional binary key (STRING)
Root reason is [HUDI-3378|https://github.com/apache/hudi/pull/7345] change
parquet reader. The latest parquet reader is compatible with spark but not
fully compatible with flink due to flink parquet schema is different from spark
parquet schema.
We will make two patch, the first patch fix this bug in 0.13.x. The last patch
fix diff schema between flink parquet and spark parquet.
> Fix flink online clustering exception when using complex type.
> --
>
> Key: HUDI-6221
> URL: https://issues.apache.org/jira/browse/HUDI-6221
> Project: Apache Hudi
> Issue Type: Bug
> Components: flink
>Reporter: Ying Lin
>Priority: Major
>
> When using flink 1.13.6 and hudi 0.13.0 cow + append + clustering mode, if
> the field list contains map type and aysnc clustering job scheduled, will
> throw exception:
> {quote}The requested schema is not compatible with the file schema.
> incompatible types: required binary key (STRING) != optional binary key
> (STRING)
> {quote}
> Root reason is [HUDI-3378|https://github.com/apache/hudi/pull/7345] change
> parquet reader. The latest parquet reader is compatible with spark but not
> fully compatible with flink due to flink parquet schema is different from
> spark parquet schema.
> We will make two patch, the first patch fix this bug in 0.13.x. The last
> patch fix diff schema between flink parquet and spark parquet.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
