snleee commented on issue #5116: Add Azure Data Lake Gen2 connector for PinotFS URL: https://github.com/apache/incubator-pinot/pull/5116#issuecomment-597961687 md5 computation time benchmark (on macbook with ssd): (md5 compute/ total time) <- total time = IO(read contents from file) + md5 hash computation 1 mb: 17 / 20 ms 10 mb: 54 / 62 ms 100 mb: 402 / 462 ms 1000 mb: 2998 / 3497 ms So, `computing md5 hash on 1GB file adds about 3 seconds.` Since this adds non-trivial overheads (depending on the use case), I updated the code to make md5 check configurable.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
