[jira] [Commented] (HUDI-1288) DeltaSync:writeToSink fails with Unknown datum type org.apache.avro.JsonProperties$Null
[
https://issues.apache.org/jira/browse/HUDI-1288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17313835#comment-17313835
]
sivabalan narayanan commented on HUDI-1288:
---
Closing out this Jira as we don't have any plans to back port fixes.
> DeltaSync:writeToSink fails with Unknown datum type
> org.apache.avro.JsonProperties$Null
> ---
>
> Key: HUDI-1288
> URL: https://issues.apache.org/jira/browse/HUDI-1288
> Project: Apache Hudi
> Issue Type: Bug
> Components: DeltaStreamer
>Reporter: Michal Swiatowy
>Priority: Major
> Labels: sev:critical, user-support-issues
>
> After updating to Hudi version 0.5.3 (prev. 0.5.2-incubating) I run into
> following error message on write to HDFS:
> {code:java}
> 2020-09-18 12:54:38,651 [Driver] INFO
> HoodieTableMetaClient:initTableAndGetMetaClient:379 - Finished initializing
> Table of type MERGE_ON_READ from
> /master_data/6FQS/hudi_test/S_INCOMINGMESSAGEDETAIL_CDC
> 2020-09-18 12:54:38,663 [Driver] INFO DeltaSync:setupWriteClient:470 -
> Setting up Hoodie Write Client
> 2020-09-18 12:54:38,695 [Driver] INFO DeltaSync:registerAvroSchemas:522 -
> Registering Schema
>
[jira] [Commented] (HUDI-1288) DeltaSync:writeToSink fails with Unknown datum type org.apache.avro.JsonProperties$Null
[
https://issues.apache.org/jira/browse/HUDI-1288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272267#comment-17272267
]
Vinoth Chandar commented on HUDI-1288:
--
https://cwiki.apache.org/confluence/display/HUDI/Release+Management talks
about this in more detail. We are not planning on doing backports, rather we
want to make rolling forward to a newer release much easier/smoother.
> DeltaSync:writeToSink fails with Unknown datum type
> org.apache.avro.JsonProperties$Null
> ---
>
> Key: HUDI-1288
> URL: https://issues.apache.org/jira/browse/HUDI-1288
> Project: Apache Hudi
> Issue Type: Bug
> Components: DeltaStreamer
>Reporter: Michal Swiatowy
>Priority: Major
> Labels: user-support-issues
>
> After updating to Hudi version 0.5.3 (prev. 0.5.2-incubating) I run into
> following error message on write to HDFS:
> {code:java}
> 2020-09-18 12:54:38,651 [Driver] INFO
> HoodieTableMetaClient:initTableAndGetMetaClient:379 - Finished initializing
> Table of type MERGE_ON_READ from
> /master_data/6FQS/hudi_test/S_INCOMINGMESSAGEDETAIL_CDC
> 2020-09-18 12:54:38,663 [Driver] INFO DeltaSync:setupWriteClient:470 -
> Setting up Hoodie Write Client
> 2020-09-18 12:54:38,695 [Driver] INFO DeltaSync:registerAvroSchemas:522 -
> Registering Schema
>
[jira] [Commented] (HUDI-1288) DeltaSync:writeToSink fails with Unknown datum type org.apache.avro.JsonProperties$Null
[
https://issues.apache.org/jira/browse/HUDI-1288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272228#comment-17272228
]
sivabalan narayanan commented on HUDI-1288:
---
[~xleesf] [~vinoth]: do we have a process to back port in general. Can either
of you respond to this ticket.
> DeltaSync:writeToSink fails with Unknown datum type
> org.apache.avro.JsonProperties$Null
> ---
>
> Key: HUDI-1288
> URL: https://issues.apache.org/jira/browse/HUDI-1288
> Project: Apache Hudi
> Issue Type: Bug
> Components: DeltaStreamer
>Reporter: Michal Swiatowy
>Priority: Major
> Labels: user-support-issues
>
> After updating to Hudi version 0.5.3 (prev. 0.5.2-incubating) I run into
> following error message on write to HDFS:
> {code:java}
> 2020-09-18 12:54:38,651 [Driver] INFO
> HoodieTableMetaClient:initTableAndGetMetaClient:379 - Finished initializing
> Table of type MERGE_ON_READ from
> /master_data/6FQS/hudi_test/S_INCOMINGMESSAGEDETAIL_CDC
> 2020-09-18 12:54:38,663 [Driver] INFO DeltaSync:setupWriteClient:470 -
> Setting up Hoodie Write Client
> 2020-09-18 12:54:38,695 [Driver] INFO DeltaSync:registerAvroSchemas:522 -
> Registering Schema
>
[jira] [Commented] (HUDI-1288) DeltaSync:writeToSink fails with Unknown datum type org.apache.avro.JsonProperties$Null
[
https://issues.apache.org/jira/browse/HUDI-1288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199882#comment-17199882
]
Michal Swiatowy commented on HUDI-1288:
---
I'm not 100% sure but I think this
org.apache.hudi.common.util.HoodieAvroUtils#rewrite method has an issue.
If you compare it with 0.6.0 branch you will notice this statement:
{code:java}
if (f.defaultVal() instanceof JsonProperties.Null) {
...
{code}
which is exactly the case.
Can we backport https://issues.apache.org/jira/browse/HUDI-803 to 0.5.3 version?
[~xleesf]
> DeltaSync:writeToSink fails with Unknown datum type
> org.apache.avro.JsonProperties$Null
> ---
>
> Key: HUDI-1288
> URL: https://issues.apache.org/jira/browse/HUDI-1288
> Project: Apache Hudi
> Issue Type: Bug
> Components: DeltaStreamer
>Reporter: Michal Swiatowy
>Priority: Major
>
> After updating to Hudi version 0.5.3 (prev. 0.5.2-incubating) I run into
> following error message on write to HDFS:
> {code:java}
> 2020-09-18 12:54:38,651 [Driver] INFO
> HoodieTableMetaClient:initTableAndGetMetaClient:379 - Finished initializing
> Table of type MERGE_ON_READ from
> /master_data/6FQS/hudi_test/S_INCOMINGMESSAGEDETAIL_CDC
> 2020-09-18 12:54:38,663 [Driver] INFO DeltaSync:setupWriteClient:470 -
> Setting up Hoodie Write Client
> 2020-09-18 12:54:38,695 [Driver] INFO DeltaSync:registerAvroSchemas:522 -
> Registering Schema
>
[jira] [Commented] (HUDI-1288) DeltaSync:writeToSink fails with Unknown datum type org.apache.avro.JsonProperties$Null
[
https://issues.apache.org/jira/browse/HUDI-1288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199078#comment-17199078
]
Michal Swiatowy commented on HUDI-1288:
---
More interestingly when I remove `default: null` occurrences from the schema
and run streamer all works fine (0.5.3 version). So I would say this boils it
down to this particular bug in Avro lib.
> DeltaSync:writeToSink fails with Unknown datum type
> org.apache.avro.JsonProperties$Null
> ---
>
> Key: HUDI-1288
> URL: https://issues.apache.org/jira/browse/HUDI-1288
> Project: Apache Hudi
> Issue Type: Bug
> Components: DeltaStreamer
>Reporter: Michal Swiatowy
>Priority: Major
>
> After updating to Hudi version 0.5.3 (prev. 0.5.2-incubating) I run into
> following error message on write to HDFS:
> {code:java}
> 2020-09-18 12:54:38,651 [Driver] INFO
> HoodieTableMetaClient:initTableAndGetMetaClient:379 - Finished initializing
> Table of type MERGE_ON_READ from
> /master_data/6FQS/hudi_test/S_INCOMINGMESSAGEDETAIL_CDC
> 2020-09-18 12:54:38,663 [Driver] INFO DeltaSync:setupWriteClient:470 -
> Setting up Hoodie Write Client
> 2020-09-18 12:54:38,695 [Driver] INFO DeltaSync:registerAvroSchemas:522 -
> Registering Schema
>
[jira] [Commented] (HUDI-1288) DeltaSync:writeToSink fails with Unknown datum type org.apache.avro.JsonProperties$Null
[
https://issues.apache.org/jira/browse/HUDI-1288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199077#comment-17199077
]
Michal Swiatowy commented on HUDI-1288:
---
Hi, yes 0.5.2 works fine.
> DeltaSync:writeToSink fails with Unknown datum type
> org.apache.avro.JsonProperties$Null
> ---
>
> Key: HUDI-1288
> URL: https://issues.apache.org/jira/browse/HUDI-1288
> Project: Apache Hudi
> Issue Type: Bug
> Components: DeltaStreamer
>Reporter: Michal Swiatowy
>Priority: Major
>
> After updating to Hudi version 0.5.3 (prev. 0.5.2-incubating) I run into
> following error message on write to HDFS:
> {code:java}
> 2020-09-18 12:54:38,651 [Driver] INFO
> HoodieTableMetaClient:initTableAndGetMetaClient:379 - Finished initializing
> Table of type MERGE_ON_READ from
> /master_data/6FQS/hudi_test/S_INCOMINGMESSAGEDETAIL_CDC
> 2020-09-18 12:54:38,663 [Driver] INFO DeltaSync:setupWriteClient:470 -
> Setting up Hoodie Write Client
> 2020-09-18 12:54:38,695 [Driver] INFO DeltaSync:registerAvroSchemas:522 -
> Registering Schema
>
[jira] [Commented] (HUDI-1288) DeltaSync:writeToSink fails with Unknown datum type org.apache.avro.JsonProperties$Null
[
https://issues.apache.org/jira/browse/HUDI-1288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198992#comment-17198992
]
leesf commented on HUDI-1288:
-
[~soltar] I found there are still some users face the issue
https://github.com/apache/avro/pull/290#issuecomment-625731714. and does
0.5.2-incubating works well?
> DeltaSync:writeToSink fails with Unknown datum type
> org.apache.avro.JsonProperties$Null
> ---
>
> Key: HUDI-1288
> URL: https://issues.apache.org/jira/browse/HUDI-1288
> Project: Apache Hudi
> Issue Type: Bug
> Components: DeltaStreamer
>Reporter: Michal Swiatowy
>Priority: Major
>
> After updating to Hudi version 0.5.3 (prev. 0.5.2-incubating) I run into
> following error message on write to HDFS:
> {code:java}
> 2020-09-18 12:54:38,651 [Driver] INFO
> HoodieTableMetaClient:initTableAndGetMetaClient:379 - Finished initializing
> Table of type MERGE_ON_READ from
> /master_data/6FQS/hudi_test/S_INCOMINGMESSAGEDETAIL_CDC
> 2020-09-18 12:54:38,663 [Driver] INFO DeltaSync:setupWriteClient:470 -
> Setting up Hoodie Write Client
> 2020-09-18 12:54:38,695 [Driver] INFO DeltaSync:registerAvroSchemas:522 -
> Registering Schema
>
