nsivabalan commented on issue #8031: URL: https://github.com/apache/hudi/issues/8031#issuecomment-1453950617
``` import java.sql.Timestamp import spark.implicits._ val df = Seq( (1, Timestamp.valueOf("2014-01-01 23:00:01"), "abc"), (1, Timestamp.valueOf("2014-11-30 12:40:32"), "abc"), (2, Timestamp.valueOf("2016-12-29 09:54:00"), "def"), (2, Timestamp.valueOf("2016-05-09 10:12:43"), "def") ).toDF("typeId","eventTime", "str") import org.apache.hudi.QuickstartUtils._ import scala.collection.JavaConversions._ import org.apache.spark.sql.SaveMode._ import org.apache.hudi.DataSourceReadOptions._ import org.apache.hudi.DataSourceWriteOptions._ import org.apache.hudi.config.HoodieWriteConfig._ import org.apache.hudi.common.model.HoodieRecord df.write.format("hudi"). option("hoodie.insert.shuffle.parallelism", "2"). option("hoodie.upsert.shuffle.parallelism", "2"). option("hoodie.datasource.write.precombine.field", "typeId"). option("hoodie.datasource.write.partitionpath.field", "eventTime"). option("hoodie.datasource.write.recordkey.field", "str"). option("hoodie.datasource.write.keygenerator.class","org.apache.hudi.keygen.TimestampBasedKeyGenerator"). option("hoodie.deltastreamer.keygen.timebased.timestamp.type","DATE_STRING"). option("hoodie.deltastreamer.keygen.timebased.timezone","GMT+8:00"). option("hoodie.deltastreamer.keygen.timebased.input.dateformat","yyyy-MM-dd hh:mm:ss"). option("hoodie.deltastreamer.keygen.timebased.output.dateformat","yyyy-MM-dd"). option("hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled","true"). option("hoodie.table.name", "hudi_tbl"). mode(Overwrite). save("/tmp/hudi_tbl_trial/") ``` ls of base path ``` ls -ltr /tmp/hudi_tbl_trial/ total 0 drwxr-xr-x 6 nsb wheel 192 Mar 3 10:40 2016-12-30 drwxr-xr-x 6 nsb wheel 192 Mar 3 10:40 2014-01-02 drwxr-xr-x 6 nsb wheel 192 Mar 3 10:40 2016-05-10 drwxr-xr-x 6 nsb wheel 192 Mar 3 10:40 2014-12-01 ``` If you prefer slash encoded ``` option("hoodie.deltastreamer.keygen.timebased.output.dateformat","yyyy/MM/dd") ``` but dir will be 3 level deep ``` ls -ltr /tmp/hudi_tbl_trial/ total 0 drwxr-xr-x 4 nsb wheel 128 Mar 3 10:42 2014 drwxr-xr-x 4 nsb wheel 128 Mar 3 10:42 2016 nsb$ ls -ltr /tmp/hudi_tbl_trial/2014/ total 0 drwxr-xr-x 3 nsb wheel 96 Mar 3 10:42 01 drwxr-xr-x 3 nsb wheel 96 Mar 3 10:42 12 nsb$ ls -ltr /tmp/hudi_tbl_trial/2014/01/ total 0 drwxr-xr-x 6 nsb wheel 192 Mar 3 10:42 02 nsb$ ls -ltr /tmp/hudi_tbl_trial/2014/01/02/ total 856 -rw-r--r-- 1 nsb wheel 434759 Mar 3 10:42 b02e5e6f-9d28-42d1-b257-3728e534d477-0_3-49-76_20230303104246958.parquet ``` Guess you were missing option("hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled","true"). https://hudi.apache.org/docs/configurations/#hoodiedatasourcewritekeygeneratorconsistentlogicaltimestampenabled-1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org