[incubator-uniffle] branch master updated: [MINOR] If there is no data flush to hdfs, return directly instead of throw exception (#406)

[roryqi]

This is an automated email from the ASF dual-hosted git repository.

roryqi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-uniffle.git


The following commit(s) were added to refs/heads/master by this push:
     new 97848c88 [MINOR]  If there is no data flush to hdfs, return directly 
instead of throw exception (#406)
97848c88 is described below

commit 97848c88d525ea18ff577bd713d7197a55c01732
Author: xianjingfeng <583872...@qq.com>
AuthorDate: Tue Dec 13 14:03:45 2022 +0800

    [MINOR]  If there is no data flush to hdfs, return directly instead of 
throw exception (#406)
    
    ### What changes were proposed in this pull request?
    <!--
    Please clarify what changes you are proposing. The purpose of this section 
is to outline the changes and how this PR fixes the issue.
    If possible, please consider writing useful notes for better and faster 
reviews in your PR. See the examples below.
      1. If you refactor some codes with changing classes, showing the class 
hierarchy will help reviewers.
      2. If you fix some SQL features, you can provide some references of other 
DBMSes.
      3. If there is design documentation, please add the link.
      4. If there is a discussion in the mailing list, please add the link.
    -->
    If there is no data flush to hdfs, return directly instead of throw 
exception as follow:
    
    ```shell
    22/12/12 17:03:34 ERROR HdfsClientReadHandler: Can't list index file in  
hdfs://xxx/application_1668750926041_8177885_1670835601240/0/155-155
    java.io.FileNotFoundException: File 
hdfs://xxx/application_1668750926041_8177885_1670835601240/0/155-155 does not 
exist.
            at 
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:795)
            at 
org.apache.hadoop.hdfs.DistributedFileSystem.access$700(DistributedFileSystem.java:106)
            at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:853)
            at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:849)
            at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
            at 
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:860)
            at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517)
            at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1557)
            at 
org.apache.uniffle.storage.handler.impl.HdfsClientReadHandler.init(HdfsClientReadHandler.java:118)
            at 
org.apache.uniffle.storage.handler.impl.HdfsClientReadHandler.readShuffleData(HdfsClientReadHandler.java:152)
            at 
org.apache.uniffle.storage.handler.impl.ComposedClientReadHandler.readShuffleData(ComposedClientReadHandler.java:93)
    ```
    
    ### Why are the changes needed?
    More friendly, and may become a bug in the future
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    No need
---
 .../uniffle/storage/handler/impl/HdfsClientReadHandler.java | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git 
a/storage/src/main/java/org/apache/uniffle/storage/handler/impl/HdfsClientReadHandler.java
 
b/storage/src/main/java/org/apache/uniffle/storage/handler/impl/HdfsClientReadHandler.java
index 59ddd9c5..259fc7ac 100644
--- 
a/storage/src/main/java/org/apache/uniffle/storage/handler/impl/HdfsClientReadHandler.java
+++ 
b/storage/src/main/java/org/apache/uniffle/storage/handler/impl/HdfsClientReadHandler.java
@@ -17,6 +17,7 @@
 
 package org.apache.uniffle.storage.handler.impl;
 
+import java.io.FileNotFoundException;
 import java.util.Comparator;
 import java.util.List;
 
@@ -110,16 +111,20 @@ public class HdfsClientReadHandler extends 
AbstractClientReadHandler {
       throw new RuntimeException("Can't get FileSystem for " + baseFolder);
     }
 
-    FileStatus[] indexFiles;
-    String failedGetIndexFileMsg = "Can't list index file in  " + baseFolder;
-
+    FileStatus[] indexFiles = null;
     try {
       // get all index files
       indexFiles = fs.listStatus(baseFolder,
           file -> file.getName().endsWith(Constants.SHUFFLE_INDEX_FILE_SUFFIX)
               && (shuffleServerId == null || 
file.getName().startsWith(shuffleServerId)));
     } catch (Exception e) {
-      LOG.error(failedGetIndexFileMsg, e);
+      if (e instanceof FileNotFoundException) {
+        LOG.info("Directory[" + baseFolder
+            + "] not found. The data may not be flushed to this directory. 
Nothing will be read.");
+      } else {
+        String failedGetIndexFileMsg = "Can't list index file in  " + 
baseFolder;
+        LOG.error(failedGetIndexFileMsg, e);
+      }
       return;
     }