kkrugler commented on code in PR #8812: URL: https://github.com/apache/pinot/pull/8812#discussion_r888291260
########## pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-standalone/src/main/java/org/apache/pinot/plugin/ingestion/batch/standalone/SegmentGenerationJobRunner.java: ########## @@ -281,7 +317,11 @@ private void submitSegmentGenTask(File localTempDir, URI inputFileURI, int seqId _outputDirFS.copyFromLocalFile(localSegmentTarFile, outputSegmentTarURI); } } catch (Exception e) { - LOGGER.error("Failed to generate Pinot segment for file - {}", inputFileURI, e); + String msg = "Failed to generate Pinot segment for file - " + inputFileURI.toString(); + _failure.compareAndSet(null, new RuntimeException(msg, e)); + + // We have to decrement the latch by the number of pending tasks. Review Comment: 1. I haven't seen `shutdownNow()` throwing exceptions due to concurrent task submission - do you have any refs for that? 2. If we don't call `shutdownNow()`, then all submitted segment generation tasks will still be executed when the finally{} code block calls `_executorService.shutdown()`, yes? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org