aznwarmonkey commented on issue #4541: URL: https://github.com/apache/hudi/issues/4541#issuecomment-1019357875
@nsivabalan removing it seems to work. the problem I see is that performance looks significantly worse with it turned off. Do you have suggestions on how to speed up the ingest process? for background each day's worth of data we have roughly 4-5 billion records which is spatiotemporal data (lat, lon, timestamp with some identifier which are set to be the key) Using straight spark `save` talkes roughly an hour, using the row write was about 20-30% slower and with it off is about 2-2.5x slower. I'll have to rerun some benchmarks to confirm actual numbers -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
