Thomas Tauber-Marshall has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17136 )
Change subject: IMPALA-10529: Fix hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build ...................................................................... IMPALA-10529: Fix hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build For start option "scratch_dirs", it only considers local filesystem as the default filesystem, regardless of the setting of DefaultFS(for a remote scratch dir, it needs to explicitly set it with the remote fs prefix). However, the function AssignQueue() would assign the queue based on not only the path string but also the default filesystem setting. For example, if scratch_dirs is set as "/tmp", the scratch dir is supposed to be in the local filesystem, but the AssignQueue() would consider it as "s3a://xxx/tmp" if a s3 path is set as the default fs. To fix this, the solution is to add a bool variable to AssignQueue() to decide whether or not to check the default fs setting when parsing the file path. For all of the scratch dirs, AssignQueue() won't check the default fs. Tests: Added a unit testcase: TmpFileMgrTest::TestSpillingWithRemoteDefaultFS. Ran and Passed TmpFileMgrTest. Change-Id: Ic07945abe65d90235aa8dea92dd3c3821a4f1f53 Reviewed-on: http://gerrit.cloudera.org:8080/17136 Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> --- M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/runtime/io/scan-range.cc M be/src/runtime/test-env.h M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc 6 files changed, 56 insertions(+), 16 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/17136 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic07945abe65d90235aa8dea92dd3c3821a4f1f53 Gerrit-Change-Number: 17136 Gerrit-PatchSet: 7 Gerrit-Owner: Yida Wu <wydbaggio...@gmail.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Thomas Tauber-Marshall <tmarsh...@cloudera.com> Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com> Gerrit-Reviewer: Yida Wu <wydbaggio...@gmail.com>
