Dan Hecht has posted comments on this change. Change subject: IMPALA-5386: Fix ReopenCachedHdfsFileHandle failure case ......................................................................
Patch Set 2: > > Did you look into the conditions which triggered the failure to > > begin with ? Is there any way to trigger similar error locally > with > > debug action or stress flag ? It would be good to add a test for > > this case. > > I know the sequence of events: > 1. File is deleted using hdfs command line > 2. Run a query over the table that has the deleted file > 3. ScanRange::Open succeeds (!!) > 4. ScanRange::Read tries hdfsRead and fails, destroys the file > handle, and reopening the file handle fails. The ScanRange's file > handle reference is now invalid, but it is also non-null. > 5. Query is aborted, leading to ScanRange::Cancel > 6. ScanRange::Cancel calls ScanRange::Close, which sees that the > file handle reference is non-null and tries to release it. The > release fails, because the file handle reference is invalid. > > The problem with reproducing this on normal Hdfs is that when a > file is deleted, the subsequent Open in #2 fails, so the query > never even has a file handle. If the Open succeeded, it is a file > handle to a local file, so POSIX guarantees that the file stays > around. I have tried modifying the code and modifying the test to > produce this sequence, but it is difficult to get this particular > combination. I'm looking at what it would take. Maybe a stress option that artificially causes hdfsRead() to appear to have failed? Given that this path doesn't get exercised, it does seem like we need some way to exercise it. -- To view, visit http://gerrit.cloudera.org:8080/7020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iee982fa5e964f6c8969b2eb7e5f3eca89e793b3a Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Joe McDonnell <joemcdonn...@cloudera.com> Gerrit-Reviewer: Dan Hecht <dhe...@cloudera.com> Gerrit-Reviewer: Joe McDonnell <joemcdonn...@cloudera.com> Gerrit-Reviewer: Michael Ho <k...@cloudera.com> Gerrit-HasComments: No
