Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24744 )
Change subject: IMPALA-15290: Retry for exists check in HadoopFsCommandLineClient ...................................................................... Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/24744/3/tests/util/hdfs_util.py File tests/util/hdfs_util.py: http://gerrit.cloudera.org:8080/#/c/24744/3/tests/util/hdfs_util.py@42 PS3, Line 42: MAX_FS_EXISTS_RETRIES = 6 > nit: How about using the retry() in tests/util/retry.py? retry() doesn't simplify the code that much: def exists(self, path): """Checks if a particular path exists. 'hdfs dfs -test -e' exits 0 if the path exists. A missing path exits non-zero with empty stderr, while a transient failure (e.g. S3 503 Slow Down) exits non-zero WITH a message on stderr. Only the latter is retried, with exponential backoff. Genuine absence returns False immediately.""" path_exists = False def check_exists(): nonlocal path_exists (status, stdout, stderr) = self._hadoop_fs_shell(['-test', '-e', path]) # Path exists, or genuinely absent (non-zero status with empty stderr): both are # definitive answers, so record the result and stop retrying. if status == 0 or not stderr.strip(): path_exists = (status == 0) return True # Non-zero status with non-empty stderr is a transient failure: retry. print('{0} exists check on {1} failed: {2}; {3}'.format( self.filesystem_type, path, stderr.strip(), stdout.strip())) return False retry(check_exists, max_attempts=MAX_FS_EXISTS_RETRIES + 1, sleep_time_s=1, backoff=2, raise_immediately=True) return path_exists Note that retry() requires the function to return False for retry and True for exit. So path_exists has to be updated by the check_exists() function instead of using the return value. -- To view, visit http://gerrit.cloudera.org:8080/24744 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I071472529ec1b923e74e960a24e19b9e70c80011 Gerrit-Change-Number: 24744 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Xuebin Su <[email protected]> Gerrit-Comment-Date: Tue, 01 Sep 2026 06:26:32 +0000 Gerrit-HasComments: Yes
