[GitHub] [incubator-dolphinscheduler] yangyichao-mango commented on issue #3643: [Bug][API] OOM occurs when uploading large files

[GitBox]

yangyichao-mango commented on issue #3643:
URL: 
https://github.com/apache/incubator-dolphinscheduler/issues/3643#issuecomment-686488879


   I think If we can implement it with streaming mode, it will be better. You 
can try it as baoqi way.
   
   
   ------------------ Original ------------------
   From: Baoqi Wu <notificati...@github.com&gt;
   Date: Thu,Sep 3,2020 5:19 PM
   To: apache/incubator-dolphinscheduler 
<incubator-dolphinschedu...@noreply.github.com&gt;
   Cc: Yichao Yang <1048262...@qq.com&gt;, Comment 
<comm...@noreply.github.com&gt;
   Subject: Re: [apache/incubator-dolphinscheduler] [Bug][API] OOM occurs when 
uploading large files (#3643)
   
   
   
   
   
    
   I think we may be able to do better for this.  for multipart files in 
springboot &amp; jetty, it should use streaming mode to not directly load the 
whole file in memory.
    
   But according to spring boot's documentation:  
https://docs.spring.io/spring-boot/docs/current/api/org/springframework/boot/autoconfigure/web/servlet/MultipartProperties.html
       `file-size-threshold specifies the size threshold after which files will 
be written to disk. The default is 0.`  
   The default is 0, so, it will parse uploaded multipart file into memory, 
instead of writing to temp file.
    
   So, I guess set the   spring.http.multipart.file-size-threshold   to a value 
like  10MB.  may avoid the OutOfMemory error
    
   spring.http.multipart.file-size-threshold=10MB
    
   Can you try whether this will reduce memory usage?
    
   —
   You are receiving this because you commented.
   Reply to this email directly, view it on GitHub, or unsubscribe.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org