Re: [I] [SUPPORT] Unstable Execution Time and Many RequestHandler WARN Logs [hudi]

[via GitHub]

danny0405 commented on issue #8100:
URL: https://github.com/apache/hudi/issues/8100#issuecomment-1759271106

   Got it, you can have a try.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] [SUPPORT] Unstable Execution Time and Many RequestHandler WARN Logs [hudi]

[via GitHub]

lovemylover042 commented on issue #8100:
URL: https://github.com/apache/hudi/issues/8100#issuecomment-1759119020

   > You can always refresh the fs view but the refreshing itself is costly.
   
   But it better than every task init local file system view. That would be 
very slowly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] [SUPPORT] Unstable Execution Time and Many RequestHandler WARN Logs [hudi]

[via GitHub]

danny0405 commented on issue #8100:
URL: https://github.com/apache/hudi/issues/8100#issuecomment-1759099553

   You can always refresh the fs view but the refreshing itself is costly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] [SUPPORT] Unstable Execution Time and Many RequestHandler WARN Logs [hudi]

[via GitHub]

lovemylover042 commented on issue #8100:
URL: https://github.com/apache/hudi/issues/8100#issuecomment-1759086656

   > Did you ever try the latest release, the fs view should perform better.
   
   Sorry but i had just upgraded to 0.10.1 from 0.8.0 recently. I will concern 
upgrading to 0.14 at the future time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] [SUPPORT] Unstable Execution Time and Many RequestHandler WARN Logs [hudi]

[via GitHub]

danny0405 commented on issue #8100:
URL: https://github.com/apache/hudi/issues/8100#issuecomment-1758753847

   Did you ever try the latest release, the fs view should perform better.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] [SUPPORT] Unstable Execution Time and Many RequestHandler WARN Logs [hudi]

[via GitHub]

lovemylover042 commented on issue #8100:
URL: https://github.com/apache/hudi/issues/8100#issuecomment-1757266624

   @danny0405 I found delta commit became so slowly because it use secondary 
filesystem view when got a bad response from remote timeline server.  I think 
the bad response was caused by compaction running at the same time and timeline 
server was behind the client. Can i  force sync local view if timeline server 
was behind the client ? 
   -
   org.apache.hudi.timeline.service.RequestHandler line 501:
   // TODO: set refreshCheck to be true when timeline server became behind 
several times or some seconds
   if (refreshCheck) {
 long beginFinalCheck = System.currentTimeMillis();
 if (isLocalViewBehind(context)) {
   String errMsg =
   "Last known instant from client was "
   + 
context.queryParam(RemoteHoodieTableFileSystemView.LAST_INSTANT_TS,
   HoodieTimeline.INVALID_INSTANT_TS)
   + " but server has the following timeline "
   + 
viewManager.getFileSystemView(context.queryParam(RemoteHoodieTableFileSystemView.BASEPATH_PARAM))
   
.getTimeline().getInstants().collect(Collectors.toList());
   throw new BadRequestResponse(errMsg);
 }
 long endFinalCheck = System.currentTimeMillis();
 finalCheckTimeTaken = endFinalCheck - beginFinalCheck;
   }
   -
   Environment Description: 
   Hudi version : 0.10.1
   Spark version : 3.0.1
   Hadoop version : 3.1.1
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org