Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14311 )
Change subject: IMPALA-8950: Add -d, -f options to hdfs copyFromLocal, put, cp ...................................................................... Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/14311/1/tests/common/impala_test_suite.py File tests/common/impala_test_suite.py: http://gerrit.cloudera.org:8080/#/c/14311/1/tests/common/impala_test_suite.py@198 PS1, Line 198: cli_client > naming: ImpalaTestSuite.cli_client doesn't tell you much about what to expe changed it to `fs_cli_client` http://gerrit.cloudera.org:8080/#/c/14311/1/tests/util/hdfs_util.py File tests/util/hdfs_util.py: http://gerrit.cloudera.org:8080/#/c/14311/1/tests/util/hdfs_util.py@217 PS1, Line 217: def copy_from_local(self, src, dst): : """Wrapper around 'hdfs dfs -copyFromLocal [-f] [-p] [-l] [-d] <localsrc> ... <dst>'. : Overwrites files by default to avoid S3 consistency issues. Specifes the '-d' option : by default, which 'Skip[s] creation of temporary file with the suffix ._COPYING_.' to : avoid extraneous copies on S3. 'src' must be either a string or a list of strings.""" : assert isinstance(src, list) or isinstance(src, basestring) : src_list = src if isinstance(src, list) else [src] : (status, stdout, stderr) = self._hadoop_fs_shell(['-copyFromLocal', '-d', '-f'] + : src_list + [dst]) : assert status == 0, '{0} copy from {1} to {2} failed: '.format(self.filesystem_type, : src, dst) + stderr + '; ' + stdout > Is it useful to keep both copy_from_local() and put()? I think that these d One argument to keep copy_from_local() is that it validates that the src is on the local filesystem, and fails otherwise. I'm not sure it's that worth keeping the function around just for that extra validation, but thought I would mention it. -- To view, visit http://gerrit.cloudera.org:8080/14311 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d45db1c00554e6fb6bcc0b552596d86d4e30144 Gerrit-Change-Number: 14311 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar <stak...@cloudera.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Joe McDonnell <joemcdonn...@cloudera.com> Gerrit-Reviewer: Sahil Takiar <stak...@cloudera.com> Gerrit-Comment-Date: Thu, 26 Sep 2019 18:49:12 +0000 Gerrit-HasComments: Yes
