gaborgsomogyi commented on code in PR #26508:
URL: https://github.com/apache/flink/pull/26508#discussion_r2060486884
##########
docs/layouts/shortcodes/generated/yarn_config_configuration.html:
##########
@@ -152,6 +152,18 @@
<td>String</td>
<td>The provided usrlib directory in remote. It should be
pre-uploaded and world-readable. Flink will use it to exclude the local usrlib
directory(i.e. usrlib/ under the parent directory of FLINK_LIB_DIR). Unlike
yarn.provided.lib.dirs, YARN will not cache it on the nodes as it is for each
application. An example could be
hdfs://$namenode_address/path/of/flink/usrlib</td>
</tr>
+ <tr>
+ <td><h5>yarn.rolled-logs.exclude-pattern</h5></td>
+ <td style="word-wrap: break-word;">"hadoopfs"</td>
+ <td>String</td>
+ <td>Java regular to exclude certain log files from rolling log
aggregation. Log files matching the defined exclude pattern will be ignored
during aggregation. If a log file matches both the include and exclude
patterns, the exclude pattern takes precedence and the file will be excluded
from aggregation.</td>
Review Comment:
s/regular to/regular expression to/
##########
flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnConfigOptions.java:
##########
@@ -401,6 +401,26 @@ public class YarnConfigOptions {
+ " Unlike yarn.provided.lib.dirs, YARN
will not cache it on the nodes as it is for each application. An example could
be "
+
"hdfs://$namenode_address/path/of/flink/usrlib");
+ public static final ConfigOption<String> ROLLED_LOGS_INCLUDE_PATTERN =
+ key("yarn.rolled-logs.include-pattern")
+ .stringType()
+ .noDefaultValue()
+ .withDescription(
+ "Java regular expression to match log file names
for inclusion in rolling log aggregation."
+ + " This regex is used by YARN’s log
aggregation mechanism to identify which log files to collect."
+ + " To enable rolling aggregation in YARN,
set the `yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds`
property in `yarn-site.xml`."
+ + " Ensure that Flink’s Log4J
configuration uses FileAppender or a compatible appender that can handle file
deletions during runtime."
+ + " The regex pattern (e.g.,
`jobmanager*`) must align with the log file names defined in the Log4J
configuration (e.g., `jobmanager.log`) to ensure all relevant files will be
aggregated.");
+
+ public static final ConfigOption<String> ROLLED_LOGS_EXCLUDE_PATTERN =
+ key("yarn.rolled-logs.exclude-pattern")
+ .stringType()
+ .noDefaultValue()
+ .withDescription(
+ "Java regular to exclude certain log files from
rolling log aggregation."
Review Comment:
s/regular to/regular expression to/
##########
docs/layouts/shortcodes/generated/yarn_config_configuration.html:
##########
@@ -152,6 +152,18 @@
<td>String</td>
<td>The provided usrlib directory in remote. It should be
pre-uploaded and world-readable. Flink will use it to exclude the local usrlib
directory(i.e. usrlib/ under the parent directory of FLINK_LIB_DIR). Unlike
yarn.provided.lib.dirs, YARN will not cache it on the nodes as it is for each
application. An example could be
hdfs://$namenode_address/path/of/flink/usrlib</td>
</tr>
+ <tr>
+ <td><h5>yarn.rolled-logs.exclude-pattern</h5></td>
+ <td style="word-wrap: break-word;">"hadoopfs"</td>
+ <td>String</td>
+ <td>Java regular to exclude certain log files from rolling log
aggregation. Log files matching the defined exclude pattern will be ignored
during aggregation. If a log file matches both the include and exclude
patterns, the exclude pattern takes precedence and the file will be excluded
from aggregation.</td>
Review Comment:
> If a log file matches both the include and exclude patterns
Optional: I would add some warning when a file hits such case. It's hard to
know what happened and why without any log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
