[GitHub] [hudi] bradleyhurley commented on issue #2068: [SUPPORT]Deltastreamer Upsert Very Slow / Never Completes After Initial Data Load

[GitBox]

bradleyhurley commented on issue #2068:
URL: https://github.com/apache/hudi/issues/2068#issuecomment-689064559


   I made some tweaks and was able to get the job to complete.
   
   - Executor Cores = 1
   - Executors = 300
   - Driver Memory = 4G
   - Executor Memory = 6G
   - spark.kryoserializer.buffer.max=512m 
   
   I am still spilling to disk, but am unsure if thats the area I should focus 
my tuning efforts.
   
   
![10_23_9_26](https://user-images.githubusercontent.com/7111357/92514833-9865d400-f1e0-11ea-9ef2-b0289ed27af1.png)
   
   **Stage 13 Details - 1.5 Hours**
   
   
![Stage_13](https://user-images.githubusercontent.com/7111357/92515136-19bd6680-f1e1-11ea-8479-9701e43b391b.png)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org