[GitHub] [hadoop] steveloughran commented on issue #1160: HADOOP-16458 LocatedFileStatusFetcher.getFileStatuses failing intermittently with s3

2019-10-01 Thread GitBox
steveloughran commented on issue #1160: HADOOP-16458 
LocatedFileStatusFetcher.getFileStatuses failing intermittently with s3
URL: https://github.com/apache/hadoop/pull/1160#issuecomment-537135148
 
 
   finally merged to trunk. Thx for review


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[GitHub] [hadoop] steveloughran commented on issue #1160: HADOOP-16458 LocatedFileStatusFetcher.getFileStatuses failing intermittently with s3

2019-09-16 Thread GitBox
steveloughran commented on issue #1160: HADOOP-16458 
LocatedFileStatusFetcher.getFileStatuses failing intermittently with s3
URL: https://github.com/apache/hadoop/pull/1160#issuecomment-531802859
 
 
   Did a full test run `-Ds3guard -Ddynamodb -Dscale -Dauth`; all failures DDB 
table delete timeouts and the (known) prune one. All good otherwise.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[GitHub] [hadoop] steveloughran commented on issue #1160: HADOOP-16458 LocatedFileStatusFetcher.getFileStatuses failing intermittently with s3

2019-09-03 Thread GitBox
steveloughran commented on issue #1160: HADOOP-16458 
LocatedFileStatusFetcher.getFileStatuses failing intermittently with s3
URL: https://github.com/apache/hadoop/pull/1160#issuecomment-527513846
 
 
   checkstyle is about an 81 char line where splitting it would make things 
less readable
   
   ```
   
./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java:255:
new InterruptedIOException("Interrupted while getting file 
statuses"): Line is longer than 80 characters (found 81). [LineLength]
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[GitHub] [hadoop] steveloughran commented on issue #1160: HADOOP-16458 LocatedFileStatusFetcher.getFileStatuses failing intermittently with s3

2019-08-28 Thread GitBox
steveloughran commented on issue #1160: HADOOP-16458 
LocatedFileStatusFetcher.getFileStatuses failing intermittently with s3
URL: https://github.com/apache/hadoop/pull/1160#issuecomment-525782579
 
 
   thanks. Think I've addressed your comments; pushing up for yetus then I'll 
commit. Tested: S3 ireland


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[GitHub] [hadoop] steveloughran commented on issue #1160: HADOOP-16458 LocatedFileStatusFetcher.getFileStatuses failing intermittently with s3

2019-07-31 Thread GitBox
steveloughran commented on issue #1160: HADOOP-16458 
LocatedFileStatusFetcher.getFileStatuses failing intermittently with s3
URL: https://github.com/apache/hadoop/pull/1160#issuecomment-516959483
 
 
   Unless this test is confusing DDB so there's no cleanup, I'd point to those 
test failures less as a sign of a problem here and more that our test runs 
still seem somehow to cause confusion, even after the tombstone fixes.
   
   `ITestS3GuardDDBRootOperations` will save the results of its scans to 
somewhere under target/ which you can use to see where a discrepancy has crept 
in.  Usually its something in s3 which isn't in s3guard: an OOB event of some 
kind


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[GitHub] [hadoop] steveloughran commented on issue #1160: HADOOP-16458 LocatedFileStatusFetcher.getFileStatuses failing intermittently with s3

2019-07-31 Thread GitBox
steveloughran commented on issue #1160: HADOOP-16458 
LocatedFileStatusFetcher.getFileStatuses failing intermittently with s3
URL: https://github.com/apache/hadoop/pull/1160#issuecomment-516865161
 
 
   Updated PR
*remove unused test suite
* rebased to trunk
* split the too-long test case in I`ITestRestrictedReadAccess` into 
separate pieces, using fields to propagate paths between them. It will still 
fail fast on the first assert failure, but as the operations and assertions 
move up the set of methods used in a LocatedFileStatusFetcher scan, that's OK. 
If an early assert fails, so will the final one
   * added some specific intercept wrappers to improve legibility.
   * javadocs
   
   No changes to production code.
   
   Tested, S3 Ireland. All well except for HADOOP-16477, which surface as I'd 
set the fs.s3a.encryption.key for the bucket. Unrelated


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[GitHub] [hadoop] steveloughran commented on issue #1160: HADOOP-16458 LocatedFileStatusFetcher.getFileStatuses failing intermittently with s3

2019-07-29 Thread GitBox
steveloughran commented on issue #1160: HADOOP-16458 
LocatedFileStatusFetcher.getFileStatuses failing intermittently with s3
URL: https://github.com/apache/hadoop/pull/1160#issuecomment-516108978
 
 
   tested: s3 ireland with `-Dparallel-tests -DtestsThreadCount=12 -Ds3guard 
-Ddynamodb -Dnonauth -Dscale`
   
   Also, check out [cloudstore Release: 
2019-07-29](https://github.com/steveloughran/cloudstore/releases/tag/tag_2019-07-29_release),
 whose `locatefiles` API call invokes this codepath and lets you see what the 
debugging is up to, including printing out the statistics of operations called.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org