[ https://issues.apache.org/jira/browse/YARN-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xie YiFan updated YARN-11644: ----------------------------- Affects Version/s: 3.3.6 > LogAggregationService can't upload log in time when application finished > ------------------------------------------------------------------------ > > Key: YARN-11644 > URL: https://issues.apache.org/jira/browse/YARN-11644 > Project: Hadoop YARN > Issue Type: Improvement > Components: log-aggregation > Affects Versions: 3.3.6 > Reporter: Xie YiFan > Assignee: Xie YiFan > Priority: Minor > Attachments: image-2024-01-10-11-03-57-553.png > > > LogAggregationService is responsible for uploading log to HDFS. It applies > thread pool to execute upload task. > The workflow of upload log as follow: > # NM construct Applicaiton object when first container of a certain > application launch, then notify LogAggregationService to init > AppLogAggregationImpl. > # LogAggregationService submit AppLogAggregationImpl to task queue > # The idle worker of thread pool pulls AppLogAggregationImpl from task queue. > # AppLogAggregationImpl do while loop to check the application state, do > upload when application finished. > Suppose the following scenario: > * LogAggregationService initialize thread pool with 4 threads. > * 4 long running applications start on this NM, so all threads are occupied > by aggregator. > * The next short application starts on this NM and quickly finish, but no > idle thread for this app to upload log. > as a result, the following applications have to wait the previous > applications finish before uploading their logs. > !image-2024-01-10-11-03-57-553.png|width=599,height=195! > h4. Solution > Change the spin behavior of AppLogAggregationImpl. If application has not > finished, just return to yield current thread and resubmit itself to executor > service. So the LogAggregationService can roll the task queue and the logs of > finished application can be uploaded immediately. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org