[GitHub] [hadoop] steveloughran commented on pull request #2732: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories.

steveloughran commented on pull request #2732: URL: https://github.com/apache/hadoop/pull/2732#issuecomment-805144616 OK, +1 from me. Merge to trunk and after a test run to branch-3.3; lets wait and see what surprises surface there -- This is an automated message from the Apache Git Serv

[GitHub] [hadoop] steveloughran commented on pull request #2732: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories.

steveloughran commented on pull request #2732: URL: https://github.com/apache/hadoop/pull/2732#issuecomment-804420292 LGTM. I'm going to make one final change, but +1 this PR pending that change anyway. Can you put the CommonCallableSupplier into `org.apache.hadoop.util.func

[GitHub] [hadoop] steveloughran commented on pull request #2732: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories.

steveloughran commented on pull request #2732: URL: https://github.com/apache/hadoop/pull/2732#issuecomment-799686604 > I added a HDFS contract test as well and that fetched me a same exception as S3A: The only reason we don't have one of those already is that it slowed down HDFS te

[GitHub] [hadoop] steveloughran commented on pull request #2732: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories.

steveloughran commented on pull request #2732: URL: https://github.com/apache/hadoop/pull/2732#issuecomment-799684654 Problem is this: https://github.com/steveloughran/hadoop/blob/s3/HADOOP-17511-auditing/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CallableSuppli

[GitHub] [hadoop] steveloughran commented on pull request #2732: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories.

steveloughran commented on pull request #2732: URL: https://github.com/apache/hadoop/pull/2732#issuecomment-799604167 Hey, I've realised that my auditing PR #2675 is going to clash on CallableSupplier changes as its taking auditing spans in, activating them before an after. Can you

[GitHub] [hadoop] steveloughran commented on pull request #2732: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories.

steveloughran commented on pull request #2732: URL: https://github.com/apache/hadoop/pull/2732#issuecomment-799570184 And abfs ``` cureDistCp [ERROR] testDistCpWithIterator(org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractSecureDistCp) Time elapsed: 504.938 s <<

[GitHub] [hadoop] steveloughran commented on pull request #2732: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories.

steveloughran commented on pull request #2732: URL: https://github.com/apache/hadoop/pull/2732#issuecomment-799554516 S3A Test failure ``` [INFO] --- [INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp [

[GitHub] [hadoop] steveloughran commented on pull request #2732: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories.

steveloughran commented on pull request #2732: URL: https://github.com/apache/hadoop/pull/2732#issuecomment-799538668 codewise, all looks good and tests seem spurious. I think what I'd like to do now is checkout and tests the s3a and abfs distp suites on my system

[GitHub] [hadoop] steveloughran commented on pull request #2732: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories.

steveloughran commented on pull request #2732: URL: https://github.com/apache/hadoop/pull/2732#issuecomment-795855716 + @mukund-thakur This is an automated message from the Apache Git Service. To respond to the message, plea