njalan commented on issue #9751:
URL: https://github.com/apache/hudi/issues/9751#issuecomment-1759783515
@ad1happy2go Below are the list count for one spark streaming micro batch:
bleow are top list opreations(**first line is list count**) for table with
hudi 0.13.1 and metadata enabled:
329 (hive/warehouse/ods_xxx.db/testing_hudi13/.hoodie/metadata/.hoodie/),
229 (hive/warehouse/ods_xxx.db/testing_hudi13/.hoodie/),
50 (hive/warehouse/ods_xxx.db/testing_hudi13/.hoodie/metadata/files/),
42
(hive/warehouse/ods_xxx.db/testing_hudi13/.hoodie/.aux/.bootstrap/.partitions/00000000-0000-0000-0000-000000000000-0_1-0-1_00000000000001.hfile/),
33 (hive/warehouse/ods_xxx.db/testing_hudi13/),
14
(hive/warehouse/ods_xxx.db/testing_hudi13/.hoodie/metadata/.hoodie/.temp/),
10
(hive/warehouse/ods_xxx.db/testing_hudi13/.hoodie/.temp/20231010140342361/),
9
(hive/warehouse/ods_xxx.db/testing_hudi13/.hoodie/.temp/20231010140158325/),
7
(hive/warehouse/ods_xxx.db/testing_hudi13/.hoodie/metadata/.hoodie/.temp/20231010140509929/),
7
(hive/warehouse/ods_xxx.db/testing_hudi13/.hoodie/metadata/.hoodie/.temp/20231010140342361/),
bleow are top list opreations(**first line is list count**) for table with
hudi 0.9 and metadata disabled:
274 (hive/warehouse/ods_xxxx.db/testing_hudi09/.hoodie/),
188
(hive/warehouse/ods_xxxx.db/testing_hudi09/.hoodie/.aux/.bootstrap/.partitions/00000000-0000-0000-0000-000000000000-0_1-0-1_00000000000001.hfile/),
48 (hive/warehouse/ods_xxxx.db/testing_hudi09/),
9
(hive/warehouse/ods_xxxx.db/testing_hudi09/.hoodie/.temp/20231010140501/),
9
(hive/warehouse/ods_xxxx.db/testing_hudi09/.hoodie/.temp/20231010140401/),
9
(hive/warehouse/ods_xxxx.db/testing_hudi09/.hoodie/.temp/20231010140301/),
9
(hive/warehouse/ods_xxxx.db/testing_hudi09/.hoodie/.temp/20231010140201/),
9
(hive/warehouse/ods_xxxx.db/testing_hudi09/.hoodie/.temp/20231010140101/),
5 (hive/warehouse/ods_xxxx.db/testing_hudi09/.hoodie/.temp/),
5 (hive/warehouse/ods_xxxx.db/testing_hudi09/.hoodie/.heartbeat/),
Is there any way the reduce the list operation? If one table can reduce 50%
list operation it can reduce workload significantly where there are thousands
of of tables with local deployed object storage cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
