[GitHub] [hadoop] steveloughran commented on pull request #4766: HADOOP-18410. S3AInputStream.unbuffer() not releasing http connections

2022-08-31 Thread GitBox
steveloughran commented on PR #4766: URL: https://github.com/apache/hadoop/pull/4766#issuecomment-1232730741 ok, i'm merging this; mukund's vote, mehakmeets comments and yetus are all happy -- This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [hadoop] steveloughran commented on pull request #4766: HADOOP-18410. S3AInputStream.unbuffer() not releasing http connections

2022-08-26 Thread GitBox
steveloughran commented on PR #4766: URL: https://github.com/apache/hadoop/pull/4766#issuecomment-1228433169 tested, s3 london -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [hadoop] steveloughran commented on pull request #4766: HADOOP-18410. S3AInputStream.unbuffer() not releasing http connections

2022-08-25 Thread GitBox
steveloughran commented on PR #4766: URL: https://github.com/apache/hadoop/pull/4766#issuecomment-1227090021 need some reviews here. this is a critical bug -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [hadoop] steveloughran commented on pull request #4766: HADOOP-18410. S3AInputStream.unbuffer() not releasing http connections

2022-08-24 Thread GitBox
steveloughran commented on PR #4766: URL: https://github.com/apache/hadoop/pull/4766#issuecomment-1225489007 either the fs tracks all open streams (weak ref map) and scans them, or each stream schedules a worker to run every few minutes which will release the stream if idle. fs is probably

[GitHub] [hadoop] steveloughran commented on pull request #4766: HADOOP-18410. S3AInputStream.unbuffer() not releasing http connections

2022-08-23 Thread GitBox
steveloughran commented on PR #4766: URL: https://github.com/apache/hadoop/pull/4766#issuecomment-1223780723 if there already is a field there, yes. i've been wondering if there was a way to do some retirement of long-lived input streams. it doesn't matter so much with vectored io or

[GitHub] [hadoop] steveloughran commented on pull request #4766: HADOOP-18410. S3AInputStream.unbuffer() not releasing http connections

2022-08-22 Thread GitBox
steveloughran commented on PR #4766: URL: https://github.com/apache/hadoop/pull/4766#issuecomment-1222517838 the last patch factors out stream draining; adds test for corner cases, especially escalation from read to abort. used in classic and prefetching streams, with unit tests.

[GitHub] [hadoop] steveloughran commented on pull request #4766: HADOOP-18410. S3AInputStream.unbuffer() not releasing http connections

2022-08-20 Thread GitBox
steveloughran commented on PR #4766: URL: https://github.com/apache/hadoop/pull/4766#issuecomment-1221329260 as this draining code is used in prefetch too, i'm going to 1. create a StreamDrainer class which the prefetch stream will also switch to; this code is fussy and I don't want

[GitHub] [hadoop] steveloughran commented on pull request #4766: HADOOP-18410. S3AInputStream.unbuffer() not releasing http connections

2022-08-19 Thread GitBox
steveloughran commented on PR #4766: URL: https://github.com/apache/hadoop/pull/4766#issuecomment-1221019821 tested: s3 london -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [hadoop] steveloughran commented on pull request #4766: HADOOP-18410. S3AInputStream.unbuffer() not releasing http connections

2022-08-19 Thread GitBox
steveloughran commented on PR #4766: URL: https://github.com/apache/hadoop/pull/4766#issuecomment-1221019682 HADOOP-18410. copy fields to variables before use in a lambda expression the key cause is that even though the fields passed in to drain() were converted to references

[GitHub] [hadoop] steveloughran commented on pull request #4766: HADOOP-18410. S3AInputStream.unbuffer() not releasing http connections

2022-08-19 Thread GitBox
steveloughran commented on PR #4766: URL: https://github.com/apache/hadoop/pull/4766#issuecomment-1220968467 The async stream draining only executes (in the other thread) if the original invoker waits for the result. this is not caused by some synchronized conflict -I made