aznwarmonkey commented on issue #4541:
URL: https://github.com/apache/hudi/issues/4541#issuecomment-1019357875


   @nsivabalan removing it seems to work. the problem I see is that performance 
looks significantly worse with it turned off. Do you have suggestions on how to 
speed up the ingest process? for background each day's worth of data we have 
roughly 4-5 billion records which is spatiotemporal data (lat, lon, timestamp 
with some identifier which are set to be the key)
   
   Using straight spark `save` talkes roughly an hour, using the row write was 
about 20-30% slower and with it off is about 2-2.5x slower. I'll have to rerun 
some benchmarks to confirm actual numbers


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


Reply via email to