[ https://issues.apache.org/jira/browse/HUDI-1288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
sivabalan narayanan updated HUDI-1288: -------------------------------------- Status: Open (was: New) > DeltaSync:writeToSink fails with Unknown datum type > org.apache.avro.JsonProperties$Null > --------------------------------------------------------------------------------------- > > Key: HUDI-1288 > URL: https://issues.apache.org/jira/browse/HUDI-1288 > Project: Apache Hudi > Issue Type: Bug > Components: DeltaStreamer > Reporter: Michal Swiatowy > Priority: Major > Labels: sev:critical, user-support-issues > > After updating to Hudi version 0.5.3 (prev. 0.5.2-incubating) I run into > following error message on write to HDFS: > {code:java} > 2020-09-18 12:54:38,651 [Driver] INFO > HoodieTableMetaClient:initTableAndGetMetaClient:379 - Finished initializing > Table of type MERGE_ON_READ from > /master_data/6FQS/hudi_test/S_INCOMINGMESSAGEDETAIL_CDC > 2020-09-18 12:54:38,663 [Driver] INFO DeltaSync:setupWriteClient:470 - > Setting up Hoodie Write Client > 2020-09-18 12:54:38,695 [Driver] INFO DeltaSync:registerAvroSchemas:522 - > Registering Schema > :[{"type":"record","name":"Value","namespace":"ARC_6FQS_W.dbo.S_INCOMINGMESSAGEDETAIL","fields":[{"name":"ID","type":"long"},{"name":"OPTIMISTICLOCK","type":{"type":"long","connect.version":1,"connect.name":"io.debezium.time.Timestamp"}},{"name":"DOCUMENTAMOUNT","type":["null",{"type":"bytes","scale":4,"precision":17,"connect.version":1,"connect.parameters":{"scale":"4","connect.decimal.precision":"17"},"connect.name":"org.apache.kafka.connect.data.Decimal","logicalType":"decimal"}],"default":null},{"name":"DOCUMENTDATE","type":["null",{"type":"long","connect.version":1,"connect.name":"io.debezium.time.Timestamp"}],"default":null},{"name":"DOCUMENTNUMBER","type":["null","string"],"default":null},{"name":"PAYMENTTYPE","type":["null","string"],"default":null},{"name":"PURCHASEORDERNUMBER","type":["null","string"],"default":null},{"name":"VALUEDATE","type":["null",{"type":"long","connect.version":1,"connect.name":"io.debezium.time.Timestamp"}],"default":null},{"name":"INCOMINGMESSAGEHEADERID","type":["null","long"],"default":null},{"name":"MESSAGETEXTID","type":["null","long"],"default":null},{"name":"DUEDATE","type":["null",{"type":"long","connect.version":1,"connect.name":"io.debezium.time.Timestamp"}],"default":null},{"name":"DEBTORASCNUMBER","type":["null","string"],"default":null},{"name":"DOCUMENTTYPE","type":["null","string"],"default":null},{"name":"NUMBEROFDUEDATES","type":["null","string"],"default":null},{"name":"DUEDATEINDICATOR","type":["null","string"],"default":null},{"name":"DISPUTECODE","type":["null","string"],"default":null},{"name":"INSTRUCTIONCODE","type":["null","string"],"default":null},{"name":"PAYMENTTERMS","type":["null","string"],"default":null},{"name":"PAYMENTCONDITION","type":["null","string"],"default":null},{"name":"DISCOUNTDAYS1","type":["null","string"],"default":null},{"name":"DISCOUNTDAYS2","type":["null","string"],"default":null},{"name":"ERRORID","type":["null","string"],"default":null},{"name":"DISCOUNTPERCENT1","type":["null",{"type":"bytes","scale":5,"precision":9,"connect.version":1,"connect.parameters":{"scale":"5","connect.decimal.precision":"9"},"connect.name":"org.apache.kafka.connect.data.Decimal","logicalType":"decimal"}],"default":null},{"name":"DISCOUNTPERCENT2","type":["null",{"type":"bytes","scale":5,"precision":9,"connect.version":1,"connect.parameters":{"scale":"5","connect.decimal.precision":"9"},"connect.name":"org.apache.kafka.connect.data.Decimal","logicalType":"decimal"}],"default":null},{"name":"DEDUCTIONAMOUNT1","type":["null",{"type":"bytes","scale":4,"precision":17,"connect.version":1,"connect.parameters":{"scale":"4","connect.decimal.precision":"17"},"connect.name":"org.apache.kafka.connect.data.Decimal","logicalType":"decimal"}],"default":null},{"name":"DEDUCTIONAMOUNT2","type":["null",{"type":"bytes","scale":4,"precision":17,"connect.version":1,"connect.parameters":{"scale":"4","connect.decimal.precision":"17"},"connect.name":"org.apache.kafka.connect.data.Decimal","logicalType":"decimal"}],"default":null},{"name":"DEDUCTIONAMOUNT3","type":["null",{"type":"bytes","scale":4,"precision":17,"connect.version":1,"connect.parameters":{"scale":"4","connect.decimal.precision":"17"},"connect.name":"org.apache.kafka.connect.data.Decimal","logicalType":"decimal"}],"default":null},{"name":"DISPUTEAMOUNT","type":["null",{"type":"bytes","scale":4,"precision":17,"connect.version":1,"connect.parameters":{"scale":"4","connect.decimal.precision":"17"},"connect.name":"org.apache.kafka.connect.data.Decimal","logicalType":"decimal"}],"default":null},{"name":"CREDITNOTENUMBER","type":["null","string"],"default":null},{"name":"DEDUCTIONCODE1","type":["null","string"],"default":null},{"name":"DEDUCTIONCODE2","type":["null","string"],"default":null},{"name":"DEDUCTIONCODE3","type":["null","string"],"default":null},{"name":"BANKCHARGESAMOUNT","type":["null",{"type":"bytes","scale":4,"precision":17,"connect.version":1,"connect.parameters":{"scale":"4","connect.decimal.precision":"17"},"connect.name":"org.apache.kafka.connect.data.Decimal","logicalType":"decimal"}],"default":null},{"name":"ERRORMESSAGETYPE","type":["null","string"],"default":null},{"name":"ERRORMESSAGEID","type":["null","string"],"default":null},{"name":"MESSAGECOMMENTID","type":["null","long"],"default":null},{"name":"PAYMENTAMOUNT","type":["null",{"type":"bytes","scale":4,"precision":17,"connect.version":1,"connect.parameters":{"scale":"4","connect.decimal.precision":"17"},"connect.name":"org.apache.kafka.connect.data.Decimal","logicalType":"decimal"}],"default":null},{"name":"CHARGEBACKAMOUNT","type":["null",{"type":"bytes","scale":4,"precision":17,"connect.version":1,"connect.parameters":{"scale":"4","connect.decimal.precision":"17"},"connect.name":"org.apache.kafka.connect.data.Decimal","logicalType":"decimal"}],"default":null},{"name":"CHARGEBACKREASONCODE","type":["null","string"],"default":null},{"name":"DISPUTEACTIONCODE","type":["null","string"],"default":null},{"name":"EDIREFERENCE","type":["null","string"],"default":null},{"name":"NOTIFYINGPARTY","type":["null","string"],"default":null},{"name":"TEXTCODE","type":["null","string"],"default":null},{"name":"LEDGERITEMID","type":["null","long"],"default":null},{"name":"FULLREFERENCE","type":["null","string"],"default":null},{"name":"__source_ts_ms","type":["null","long"],"default":null},{"name":"__source_event_serial_no","type":["null","long"],"default":null},{"name":"__op","type":["null","string"],"default":null},{"name":"_hoodie_is_deleted","type":["null","boolean"],"default":null}],"connect.name":"ARC_6FQS_W.dbo.S_INCOMINGMESSAGEDETAIL.Value"}, > > {"type":"record","name":"Value","namespace":"ARC_6FQS_W.dbo.S_INCOMINGMESSAGEDETAIL","fields":[{"name":"ID","type":"long"},{"name":"OPTIMISTICLOCK","type":{"type":"long","connect.version":1,"connect.name":"io.debezium.time.Timestamp"}},{"name":"DOCUMENTAMOUNT","type":["null",{"type":"bytes","scale":4,"precision":17,"connect.version":1,"connect.parameters":{"scale":"4","connect.decimal.precision":"17"},"connect.name":"org.apache.kafka.connect.data.Decimal","logicalType":"decimal"}],"default":null},{"name":"DOCUMENTDATE","type":["null",{"type":"long","connect.version":1,"connect.name":"io.debezium.time.Timestamp"}],"default":null},{"name":"DOCUMENTNUMBER","type":["null","string"],"default":null},{"name":"PAYMENTTYPE","type":["null","string"],"default":null},{"name":"PURCHASEORDERNUMBER","type":["null","string"],"default":null},{"name":"VALUEDATE","type":["null",{"type":"long","connect.version":1,"connect.name":"io.debezium.time.Timestamp"}],"default":null},{"name":"INCOMINGMESSAGEHEADERID","type":["null","long"],"default":null},{"name":"MESSAGETEXTID","type":["null","long"],"default":null},{"name":"DUEDATE","type":["null",{"type":"long","connect.version":1,"connect.name":"io.debezium.time.Timestamp"}],"default":null},{"name":"DEBTORASCNUMBER","type":["null","string"],"default":null},{"name":"DOCUMENTTYPE","type":["null","string"],"default":null},{"name":"NUMBEROFDUEDATES","type":["null","string"],"default":null},{"name":"DUEDATEINDICATOR","type":["null","string"],"default":null},{"name":"DISPUTECODE","type":["null","string"],"default":null},{"name":"INSTRUCTIONCODE","type":["null","string"],"default":null},{"name":"PAYMENTTERMS","type":["null","string"],"default":null},{"name":"PAYMENTCONDITION","type":["null","string"],"default":null},{"name":"DISCOUNTDAYS1","type":["null","string"],"default":null},{"name":"DISCOUNTDAYS2","type":["null","string"],"default":null},{"name":"ERRORID","type":["null","string"],"default":null},{"name":"DISCOUNTPERCENT1","type":["null",{"type":"bytes","scale":5,"precision":9,"connect.version":1,"connect.parameters":{"scale":"5","connect.decimal.precision":"9"},"connect.name":"org.apache.kafka.connect.data.Decimal","logicalType":"decimal"}],"default":null},{"name":"DISCOUNTPERCENT2","type":["null",{"type":"bytes","scale":5,"precision":9,"connect.version":1,"connect.parameters":{"scale":"5","connect.decimal.precision":"9"},"connect.name":"org.apache.kafka.connect.data.Decimal","logicalType":"decimal"}],"default":null},{"name":"DEDUCTIONAMOUNT1","type":["null",{"type":"bytes","scale":4,"precision":17,"connect.version":1,"connect.parameters":{"scale":"4","connect.decimal.precision":"17"},"connect.name":"org.apache.kafka.connect.data.Decimal","logicalType":"decimal"}],"default":null},{"name":"DEDUCTIONAMOUNT2","type":["null",{"type":"bytes","scale":4,"precision":17,"connect.version":1,"connect.parameters":{"scale":"4","connect.decimal.precision":"17"},"connect.name":"org.apache.kafka.connect.data.Decimal","logicalType":"decimal"}],"default":null},{"name":"DEDUCTIONAMOUNT3","type":["null",{"type":"bytes","scale":4,"precision":17,"connect.version":1,"connect.parameters":{"scale":"4","connect.decimal.precision":"17"},"connect.name":"org.apache.kafka.connect.data.Decimal","logicalType":"decimal"}],"default":null},{"name":"DISPUTEAMOUNT","type":["null",{"type":"bytes","scale":4,"precision":17,"connect.version":1,"connect.parameters":{"scale":"4","connect.decimal.precision":"17"},"connect.name":"org.apache.kafka.connect.data.Decimal","logicalType":"decimal"}],"default":null},{"name":"CREDITNOTENUMBER","type":["null","string"],"default":null},{"name":"DEDUCTIONCODE1","type":["null","string"],"default":null},{"name":"DEDUCTIONCODE2","type":["null","string"],"default":null},{"name":"DEDUCTIONCODE3","type":["null","string"],"default":null},{"name":"BANKCHARGESAMOUNT","type":["null",{"type":"bytes","scale":4,"precision":17,"connect.version":1,"connect.parameters":{"scale":"4","connect.decimal.precision":"17"},"connect.name":"org.apache.kafka.connect.data.Decimal","logicalType":"decimal"}],"default":null},{"name":"ERRORMESSAGETYPE","type":["null","string"],"default":null},{"name":"ERRORMESSAGEID","type":["null","string"],"default":null},{"name":"MESSAGECOMMENTID","type":["null","long"],"default":null},{"name":"PAYMENTAMOUNT","type":["null",{"type":"bytes","scale":4,"precision":17,"connect.version":1,"connect.parameters":{"scale":"4","connect.decimal.precision":"17"},"connect.name":"org.apache.kafka.connect.data.Decimal","logicalType":"decimal"}],"default":null},{"name":"CHARGEBACKAMOUNT","type":["null",{"type":"bytes","scale":4,"precision":17,"connect.version":1,"connect.parameters":{"scale":"4","connect.decimal.precision":"17"},"connect.name":"org.apache.kafka.connect.data.Decimal","logicalType":"decimal"}],"default":null},{"name":"CHARGEBACKREASONCODE","type":["null","string"],"default":null},{"name":"DISPUTEACTIONCODE","type":["null","string"],"default":null},{"name":"EDIREFERENCE","type":["null","string"],"default":null},{"name":"NOTIFYINGPARTY","type":["null","string"],"default":null},{"name":"TEXTCODE","type":["null","string"],"default":null},{"name":"LEDGERITEMID","type":["null","long"],"default":null},{"name":"FULLREFERENCE","type":["null","string"],"default":null},{"name":"__source_ts_ms","type":["null","long"],"default":null},{"name":"__source_event_serial_no","type":["null","long"],"default":null},{"name":"__op","type":["null","string"],"default":null},{"name":"_hoodie_is_deleted","type":["null","boolean"],"default":null}],"connect.name":"ARC_6FQS_W.dbo.S_INCOMINGMESSAGEDETAIL.Value"}] > ... > 2020-09-18 12:54:46,106 [Driver] INFO > HoodieCopyOnWriteTable:assignInserts:681 - Total insert buckets for partition > path => [WorkloadStat {bucketNumber=0, weight=1.0}] > 2020-09-18 12:54:46,106 [Driver] INFO > HoodieCopyOnWriteTable:<init>:584 - Total Buckets :1, buckets info > => {0=BucketInfo {bucketType=INSERT, > fileIdPrefix=6b71e0b8-8cf4-40d8-8aa6-f0f9a31a1709}}, > Partition to insert buckets => {=[WorkloadStat {bucketNumber=0, > weight=1.0}]}, > UpdateLocations mapped to buckets =>{} > 2020-09-18 12:54:46,160 [Driver] INFO > AbstractHoodieWriteClient:commitOnAutoCommit:126 - Auto commit disabled for > 20200918125441 > 2020-09-18 12:54:52,075 [Driver] ERROR DeltaSync:writeToSink:410 - Delta Sync > found errors when writing. Errors/Total=27530/27530 > 2020-09-18 12:54:52,076 [Driver] ERROR DeltaSync:writeToSink:411 - Printing > out the top 100 errors > 2020-09-18 12:54:52,391 [Driver] ERROR DeltaSync:lambda$writeToSink$5:413 - > Global error : > 2020-09-18 12:54:52,392 [Driver] TRACE DeltaSync:lambda$null$4:415 - Error > for key:HoodieKey { recordKey=161195 partitionPath=} is > org.apache.avro.AvroRuntimeException: Unknown datum type > org.apache.avro.JsonProperties$Null: > org.apache.avro.JsonProperties$Null@1c95faea > 2020-09-18 12:54:52,392 [Driver] TRACE DeltaSync:lambda$null$4:415 - Error > for key:HoodieKey { recordKey=371271 partitionPath=} is > org.apache.avro.AvroRuntimeException: Unknown datum type > org.apache.avro.JsonProperties$Null: > org.apache.avro.JsonProperties$Null@1c95faea > 2020-09-18 12:54:52,392 [Driver] TRACE DeltaSync:lambda$null$4:415 - Error > for key:HoodieKey { recordKey=315920 partitionPath=} is > org.apache.avro.AvroRuntimeException: Unknown datum type > org.apache.avro.JsonProperties$Null: > org.apache.avro.JsonProperties$Null@1c95faea > 2020-09-18 12:54:52,392 [Driver] TRACE DeltaSync:lambda$null$4:415 - Error > for key:HoodieKey { recordKey=130933 partitionPath=} is > org.apache.avro.AvroRuntimeException: Unknown datum type > org.apache.avro.JsonProperties$Null: > org.apache.avro.JsonProperties$Null@1c95faea > {code} > After investigation I stumbled upon > https://issues.apache.org/jira/browse/AVRO-1954 -definitely related. > When I changed Avro schema (removed all default: null occurrences) streamer > passes. > Unfortunately that's not case for me since schema comes from Schema Registry. -- This message was sent by Atlassian Jira (v8.3.4#803005)