[GitHub] [hudi] jtmzheng commented on issue #2408: [SUPPORT] OutOfMemory on upserting into MOR dataset

2021-02-06 Thread GitBox
jtmzheng commented on issue #2408: URL: https://github.com/apache/hudi/issues/2408#issuecomment-774513919 @nsivabalan I have not encountered the issue again after temporarily lowering `hoodie.commits.archival.batch` which cleared out the large commit files being loaded for archival. I

[GitHub] [hudi] jtmzheng commented on issue #2408: [SUPPORT] OutOfMemory on upserting into MOR dataset

2021-01-11 Thread GitBox
jtmzheng commented on issue #2408: URL: https://github.com/apache/hudi/issues/2408#issuecomment-758360941 Thanks Udit! I'd tried setting `hoodie.commits.archival.batch` to 5 earlier today after going through the source code - that got my application back and running again. The

[GitHub] [hudi] jtmzheng commented on issue #2408: [SUPPORT] OutOfMemory on upserting into MOR dataset

2021-01-07 Thread GitBox
jtmzheng commented on issue #2408: URL: https://github.com/apache/hudi/issues/2408#issuecomment-756289589 Thank you, I have also tried setting `hoodie.write.buffer.limit.bytes` as per https://github.com/apache/hudi/issues/1491#issuecomment-610626104 but it still OOMs in same way (this