Yida Wu has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17711
Change subject: IMPALA-10429 Add Support for Spilling to HDFS Path Parsing ...................................................................... IMPALA-10429 Add Support for Spilling to HDFS Path Parsing We support the HDFS scratch space, but only as a test-only feature with a fixed HDFS default local path. In this patch, we extend the HDFS scratch space to support the customer's input. For supporting the function, we add a new format for HDFS scratch space path. It forces the HDFS path to have the port number to solve the contradiction to the current format of the scratch space path. For example, previously, the format for scratch space path is, take s3 for example, s3a://bucketpath:#bytes:#priority. In this case, the bucketpath doesn't have a port number. In the patch, the new format of HDFS scratch path is hdfs://ipaddr:#port:#bytes:#priority. The port number is required, therefore, there must be at least one colon in the HDFS path, the bytes and priority are optional as before. For other scratch spaces, the path format doesn’t change. The patch also adds a logic to try fetching the connection of a remote scratch space in the initialization of TmpFileMgr, in order to expose the case at the beginning if a wrong remote path is configured. Tests: Added and passed TmpFileMgrTest::TestDirectoryLimitParsingRemotePath. Ran the Core tests. Change-Id: I26393922811137278046bf82d4988ab832944f02 --- M be/src/runtime/bufferpool/buffer-pool-test.cc M be/src/runtime/io/disk-io-mgr-test.cc M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h M tests/custom_cluster/test_scratch_disk.py 6 files changed, 191 insertions(+), 74 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/17711/1 -- To view, visit http://gerrit.cloudera.org:8080/17711 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I26393922811137278046bf82d4988ab832944f02 Gerrit-Change-Number: 17711 Gerrit-PatchSet: 1 Gerrit-Owner: Yida Wu <wydbaggio...@gmail.com>
