Yida Wu has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17720
Change subject: IMPALA-10429 Add Support for Spilling to HDFS Path Parsing ...................................................................... IMPALA-10429 Add Support for Spilling to HDFS Path Parsing We support the HDFS scratch space, but only as a test-only feature with a fixed HDFS default local path. In this patch, we extend the HDFS scratch space to support the customer's input. For supporting the function, we add a new format for HDFS scratch space path. It forces the HDFS path to have the port number to solve the contradiction to the current format of the scratch space path. For example, previously, the format for scratch space path is, take s3 for example, s3a://bucketpath:#bytes:#priority. In this case, the bucketpath doesn't have a port number. In the patch, the new format of HDFS scratch path is hdfs://ipaddr:#port:#bytes:#priority. The port number is required, therefore, there must be at least one colon in the HDFS path, the bytes and priority are optional as before. For other scratch spaces, the path format doesn’t change. Tests: Added and passed TmpFileMgrTest::TestDirectoryLimitParsingRemotePath. Ran the Core tests. Change-Id: I0882ed1e80b02724dd5cb3cdb1fa7b6c2debcbf4 --- M be/src/runtime/bufferpool/buffer-pool-test.cc M be/src/runtime/io/disk-io-mgr-test.cc M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h M tests/custom_cluster/test_scratch_disk.py 6 files changed, 210 insertions(+), 63 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/17720/1 -- To view, visit http://gerrit.cloudera.org:8080/17720 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I0882ed1e80b02724dd5cb3cdb1fa7b6c2debcbf4 Gerrit-Change-Number: 17720 Gerrit-PatchSet: 1 Gerrit-Owner: Yida Wu <wydbaggio...@gmail.com>
