Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17171 )
Change subject: IMPALA-10579: Fix usage of RemoteIterator in FileSystemUtil ...................................................................... IMPALA-10579: Fix usage of RemoteIterator in FileSystemUtil HDFS FileSystem provides a listStatusIterator() API for listing remote storage using a RemoteIterator. We use it to list files when loading table file metadata. It's not guaranteed that a RemoteIterator can survive when its hasNext() or next() throws IOExceptions. We should stop the loop in this case. Otherwise, we may go into a infinite loop. Without HADOOP-16685, it's also not guaranteed that FileSystem.listStatusIterator() will throw a FileNotFoundException when the path doesn't exist. This patch refactors the file listing iterators so we don't need to depend on these two assumptions. The basic idea is: - On one side, we should not depends on other RemoteIterator's behavior after exception. - On the other side, we try to make our own iterators more robust on transient sub-directories. So table loading won't be failed by them. Tests: - Loop test_insert_stress.py 100 times. Verified the non-existing subdirs are skipped and inserts are stable in a high concurrency. Change-Id: I859bd4f976c51a34eb6a03cefd2ddcdf11656cea Reviewed-on: http://gerrit.cloudera.org:8080/17171 Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> --- M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java 1 file changed, 83 insertions(+), 32 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/17171 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I859bd4f976c51a34eb6a03cefd2ddcdf11656cea Gerrit-Change-Number: 17171 Gerrit-PatchSet: 7 Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Kurt Deschler <kdesc...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Vihang Karajgaonkar <vih...@cloudera.com>
