[GitHub] [hudi] BohanZhang0222 commented on issue #8902: [SUPPORT] Flink Async Compaction MOR Table，OutOfMemoryError: Requested array size exceeds VM limit

2023-06-11 Thread via GitHub 


BohanZhang0222 commented on issue #8902:
URL: https://github.com/apache/hudi/issues/8902#issuecomment-1586498323

   > 
   
   
   > It seems the filesystem view takes too much memory. It starts on the 
client machine that you submit the job.
   
   how to adjust client memory ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hudi] BohanZhang0222 commented on issue #8902: [SUPPORT] Flink Async Compaction MOR Table，OutOfMemoryError: Requested array size exceeds VM limit

2023-06-07 Thread via GitHub 


BohanZhang0222 commented on issue #8902:
URL: https://github.com/apache/hudi/issues/8902#issuecomment-1581964681

   > There is an option for the sort memory, did you try that, did you try to 
turn the memory for flink JM?
   
   1. Can you provide specific option? Thanks.
   2. My start command: `./bin/flink run -ynm 
compaction_ods_ssp_cloud_vss_on_changed_topic_hudi -yqu eps -ytm 8192 -yd -m 
yarn-cluster -yt /chj/flink/flink-1.14.0/sql_jar -C 
file:///chj/flink/flink-1.14.0/sql_jar/juicefs-hadoop-1.0.0-lixiang-dip.jar -c 
org.apache.hudi.sink.compact.HoodieFlinkCompactor  
/chj/flink/flink-1.14.0/sql_jar/hudi-flink1.14-bundle-0.13.0.jar --path 
jfs:// --compaction-max-memory 1024 --seq LIFO --compaction-tasks 10 
--min-compaction-interval-seconds 120 --service`  FileSystemViewManager Server 
Run on the host machine instead of JM.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org