[jira] [Commented] (FLINK-28283) Improving the log of flink when job start and deploy
[ https://issues.apache.org/jira/browse/FLINK-28283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686987#comment-17686987 ] zlzhang0122 commented on FLINK-28283: - [~xtsong] Right you are, actually what I think is cache theses logs for some short time, but indeed this may cause some problem such as timeliness and lose of logs, so this is fine with me to close this ticket. [~chesnay] Sure, we can hiding the log message via log4j filters. > Improving the log of flink when job start and deploy > > > Key: FLINK-28283 > URL: https://issues.apache.org/jira/browse/FLINK-28283 > Project: Flink > Issue Type: Improvement > Components: Runtime / Task >Affects Versions: 1.14.2 >Reporter: zlzhang0122 >Priority: Major > > When running a large job with many operators and subtasks on flink, the > JobManager and TaskManager will have a huge logs about the subtask executing > msg such as "XXX switched from CREATED to SCHEDULED、XXX switched from > SCHEDULED to DEPLOYING 、XXX switched from DEPLOYING to RUNNING 、XXX switched > from RUNNING to CANCELING、XXX switched from CANCELING to CANCELED", etc. . > Maybe we can do some improvement about this, such as aggregate these msg to > reduce the log, or change the log level and only logs the failure msg and > subtask, etc. Not so sure about the solution, but these msg is really too > much. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-28283) Improving the log of flink when job start and deploy
[ https://issues.apache.org/jira/browse/FLINK-28283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686825#comment-17686825 ] Xintong Song commented on FLINK-28283: -- I'm also inclined to close the ticket. But let's wait a bit longer for the response from the reporter. > Improving the log of flink when job start and deploy > > > Key: FLINK-28283 > URL: https://issues.apache.org/jira/browse/FLINK-28283 > Project: Flink > Issue Type: Improvement > Components: Runtime / Task >Affects Versions: 1.14.2 >Reporter: zlzhang0122 >Priority: Major > > When running a large job with many operators and subtasks on flink, the > JobManager and TaskManager will have a huge logs about the subtask executing > msg such as "XXX switched from CREATED to SCHEDULED、XXX switched from > SCHEDULED to DEPLOYING 、XXX switched from DEPLOYING to RUNNING 、XXX switched > from RUNNING to CANCELING、XXX switched from CANCELING to CANCELED", etc. . > Maybe we can do some improvement about this, such as aggregate these msg to > reduce the log, or change the log level and only logs the failure msg and > subtask, etc. Not so sure about the solution, but these msg is really too > much. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-28283) Improving the log of flink when job start and deploy
[ https://issues.apache.org/jira/browse/FLINK-28283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686405#comment-17686405 ] Chesnay Schepler commented on FLINK-28283: -- I'm inclined to close this ticket; aggregations won't work, and hiding the log message can already be achieved via log4j filters. > Improving the log of flink when job start and deploy > > > Key: FLINK-28283 > URL: https://issues.apache.org/jira/browse/FLINK-28283 > Project: Flink > Issue Type: Improvement > Components: Runtime / Task >Affects Versions: 1.14.2 >Reporter: zlzhang0122 >Priority: Major > > When running a large job with many operators and subtasks on flink, the > JobManager and TaskManager will have a huge logs about the subtask executing > msg such as "XXX switched from CREATED to SCHEDULED、XXX switched from > SCHEDULED to DEPLOYING 、XXX switched from DEPLOYING to RUNNING 、XXX switched > from RUNNING to CANCELING、XXX switched from CANCELING to CANCELED", etc. . > Maybe we can do some improvement about this, such as aggregate these msg to > reduce the log, or change the log level and only logs the failure msg and > subtask, etc. Not so sure about the solution, but these msg is really too > much. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-28283) Improving the log of flink when job start and deploy
[ https://issues.apache.org/jira/browse/FLINK-28283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686338#comment-17686338 ] Xintong Song commented on FLINK-28283: -- I think logging all the task state changes might be necessary. - We cannot aggregate these logs, because state changing of tasks are independent from each other. - We cannot only log failures, because tasks may stuck in some state and never reaches fail / running state. In such cases, we need to know what state the task is in. Same for changing log level. > Improving the log of flink when job start and deploy > > > Key: FLINK-28283 > URL: https://issues.apache.org/jira/browse/FLINK-28283 > Project: Flink > Issue Type: Improvement > Components: Runtime / Task >Affects Versions: 1.14.2 >Reporter: zlzhang0122 >Priority: Major > > When running a large job with many operators and subtasks on flink, the > JobManager and TaskManager will have a huge logs about the subtask executing > msg such as "XXX switched from CREATED to SCHEDULED、XXX switched from > SCHEDULED to DEPLOYING 、XXX switched from DEPLOYING to RUNNING 、XXX switched > from RUNNING to CANCELING、XXX switched from CANCELING to CANCELED", etc. . > Maybe we can do some improvement about this, such as aggregate these msg to > reduce the log, or change the log level and only logs the failure msg and > subtask, etc. Not so sure about the solution, but these msg is really too > much. -- This message was sent by Atlassian Jira (v8.20.10#820010)
