Re: [I] [SUPPORT] Performance Tuning: Slow stages (Building Workload Profile & Getting Small files from partitions) during Hudi Writes [hudi]

2024-03-08 Thread via GitHub 


FFCMSouza commented on issue #2620:
URL: https://github.com/apache/hudi/issues/2620#issuecomment-1986433311

   I'm having the same problema on hudi version 0.14.1 and spark 3.4.1.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] [SUPPORT] Performance Tuning: Slow stages (Building Workload Profile & Getting Small files from partitions) during Hudi Writes [hudi]

2023-11-16 Thread via GitHub 


zyclove commented on issue #2620:
URL: https://github.com/apache/hudi/issues/2620#issuecomment-1815661842

   I also encountered the same problem with 0.14.0, how to solve it?
   disable metadata ?
   set hoodie.metadata.table=false;
   
   change hoodie.parquet.small.file.limit ?
   
   set hoodie.bloom.index.prune.by.ranges = false ?
   
   change hoodie.memory.merge.max.size ?
   
   Can this be optimized in hudi 1.0? This stage is simply too time consuming.
   
   
![image](https://github.com/apache/hudi/assets/15028279/090b9a46-94fc-4d8c-9ea8-041437632761)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org