Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19869 )
Change subject: IMPALA-12123: Fix crash triggered by incomplete HDFS cache reads ...................................................................... IMPALA-12123: Fix crash triggered by incomplete HDFS cache reads When using HDFS caching, the HDFS cache may not have the full buffer in memory, and it can return a buffer that is incomplete. In this case, the code falls back to the ordinary read path. However, the ScanRange cache_ structure is still set up, and the code in ScanRange::ReadSubRanges() tries to use it. This can crash, because the buffer is too short (and may have been freed). This changes the code to null out the cache_ data structure when there is an incomplete read from the HDFS cache. Testing: - Reproduced the crash stack manually by putting a Parquet file with a page index in HDFS cache and manually forcing it down the incomplete read codepath. - Modified the disk-io-mgr-test and CacheReaderTestStub to simulate the incomplete read case. The test will hit a DCHECK or crash without this fixup. Change-Id: I51d8be6c03716badee81675447ed94ae6249b21b Reviewed-on: http://gerrit.cloudera.org:8080/19869 Reviewed-by: Michael Smith <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> Reviewed-by: Zoltan Borok-Nagy <[email protected]> --- M be/src/runtime/io/cache-reader-test-stub.h M be/src/runtime/io/disk-io-mgr-test.cc M be/src/runtime/io/scan-range.cc 3 files changed, 87 insertions(+), 47 deletions(-) Approvals: Michael Smith: Looks good to me, but someone else must approve Impala Public Jenkins: Verified Zoltan Borok-Nagy: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/19869 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I51d8be6c03716badee81675447ed94ae6249b21b Gerrit-Change-Number: 19869 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
