Re: [I] [SUPPORT] Hudi 0.14.0 - deletion from table failing for org.apache.hudi.keygen.TimestampBasedKeyGenerator [hudi]
Priyanka128 commented on issue #10823: URL: https://github.com/apache/hudi/issues/10823#issuecomment-2111792611 > I think your timestamp.type should be "DATE_STRING". Tried this but getting below exception: _Caused by: java.lang.RuntimeException: hoodie.keygen.timebased.timestamp.scalar.time.unit is not specified but scalar it supplied as time value at org.apache.hudi.keygen.TimestampBasedAvroKeyGenerator.convertLongTimeToMillis(TimestampBasedAvroKeyGenerator.java:216) at org.apache.hudi.keygen.TimestampBasedAvroKeyGenerator.getPartitionPath(TimestampBasedAvroKeyGenerator.java:187) at org.apache.hudi.keygen.TimestampBasedAvroKeyGenerator.getPartitionPath(TimestampBasedAvroKeyGenerator.java:118) ... 18 more_ After encountering this exception, removed "hoodie.deltastreamer.keygen.timebased.timestamp.scalar.time.unit" -> "DAYS" but still same exception "Caused by: java.lang.RuntimeException: hoodie.keygen.timebased.timestamp.scalar.time.unit is not specified but scalar it supplied as time value" was coming. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT] Hudi 0.14.0 - deletion from table failing for org.apache.hudi.keygen.TimestampBasedKeyGenerator [hudi]
xicm commented on issue #10823: URL: https://github.com/apache/hudi/issues/10823#issuecomment-2024834691 I think your timestamp.type should be "DATE_STRING". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT] Hudi 0.14.0 - deletion from table failing for org.apache.hudi.keygen.TimestampBasedKeyGenerator [hudi]
ad1happy2go commented on issue #10823:
URL: https://github.com/apache/hudi/issues/10823#issuecomment-1980552281
@ShrutiBansal309 Able to reproduce this issue. Issue comes even when we just
try to read this table.
JIRA - https://issues.apache.org/jira/browse/HUDI-7485
Reproducible Code
```
columns = ["ts","uuid","rider","driver","fare","dt"]
data
=[(1695159649087,"334e26e9-8355-45cc-97c6-c31daf0df330","rider-A","driver-K",19.10,"2012-01-01"),
(1695091554788,"e96c4396-3fad-413a-a942-4cb36106d721","rider-B","driver-L",27.70
,"2012-01-01"),
(1695046462179,"9909a8b1-2d15-4d3d-8ec9-efc48c536a00","rider-C","driver-M",33.90
,"2012-01-01"),
(1695516137016,"e3cf430c-889d-4015-bc98-59bdce1e530c","rider-C","driver-N",34.15,"2012-01-01")]
inserts = spark.createDataFrame(data).toDF(*columns)
hudi_options = {
'hoodie.table.name': tableName,
'hoodie.datasource.write.recordkey.field' : 'uuid',
'hoodie.datasource.write.precombine.field' : 'ts',
'hoodie.datasource.write.partitionpath.field': 'dt',
'hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled' :
'true',
'hoodie.datasource.write.keygenerator.class' :
'org.apache.hudi.keygen.TimestampBasedKeyGenerator',
'hoodie.keygen.timebased.timestamp.type' : 'SCALAR',
'hoodie.keygen.timebased.timestamp.scalar.time.unit' : 'DAYS',
'hoodie.keygen.timebased.input.dateformat' : '-MM-dd',
'hoodie.keygen.timebased.output.dateformat' : '-MM-dd',
'hoodie.keygen.timebased.timezone' : 'GMT+8:00',
'hoodie.datasource.write.hive_style_partitioning' : 'true',
}
# Insert data
inserts.withColumn("dt", expr("CAST(dt as date)")).write.format("hudi"). \
options(**hudi_options). \
mode("overwrite"). \
save(basePath)
deleteDF=spark.read.format("hudi").load(basePath)
deleteDF.show()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
