This is an automated email from the ASF dual-hosted git repository.
yuqi1129 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new 35c942066c [#13274] fix(log): keep every rolled log archive and bound
log retention (#13275)
35c942066c is described below
commit 35c942066cc161766e39a1714d6e29affddcab14
Author: Jerry Shao <[email protected]>
AuthorDate: Fri Sep 18 14:50:44 2026 +0800
[#13274] fix(log): keep every rolled log archive and bound log retention
(#13275)
### What changes were proposed in this pull request?
- Log4j2 templates (server, Iceberg REST, Lance REST): add `%i` to every
size-rolled archive name, set
`fileIndex = nomax`, and scope each `Delete` action to the log's own
archives (`IfFileName`,
`maxDepth = 1`). An archive is deleted when it is older than `logMaxAge`
(30d), or, oldest first, when
the log's archives exceed its total size cap. The caps are properties at
the top of each file: server 2GB,
audit 10GB, lineage 1GB, Iceberg REST 2GB, Lance REST 1GB. The roll size
goes from 10MB to 100MB;
lineage now also rolls by size.
- `.out` files: `bin/common.sh.template` adds `rotate_out_file`. Every
`start` rotates the file and keeps
`GRAVITINO_OUT_FILE_KEEP` (default 5) previous files.
- Helm charts (gravitino, iceberg-rest-server, lance-rest-server): the
same retention for the main appender,
with new values `logMaxAge`, `serverLogMaxTotalSize`,
`rollingAppenderStrategyFileIndex`, and
`rollingAppenderStrategyDeleteIfFileNameGlob`.
- Docs: a "Log rotation and retention" section in `how-to-install.md`,
and the audit retention example in
`gravitino-server-config.md`.
### Why are the changes needed?
Each 10MB roll overwrites the previous archive of the day, so most of
each day's logs are lost. The audit
log keeps at most 7 archives a day, `Delete` actions remove unrelated
old files under the log directory,
no log has a size limit, and `.out` files grow forever.
Fix: #13274
### Does this PR introduce _any_ user-facing change?
- Archive names become `<log>_<yyyyMMdd>.<index>.log.gz`, and the roll
size is 100MB.
- New log4j2 properties: `logMaxAge`, `serverLogMaxTotalSize`,
`auditLogMaxTotalSize`, `lineageLogMaxTotalSize`.
- New env var `GRAVITINO_OUT_FILE_KEEP`.
- New Helm values listed above. The chart defaults change the same way
as the tarball.
- `appender.audit_file.strategy.delete.ifAll.ifLastModified.age` no
longer exists. Log4j2 rejects a
configuration file that still sets it; the docs describe the
replacement.
### How was this patch tested?
- `TestLog4j2RollingConfiguration` (10 cases): loads each template with
a 1KB roll size and checks that no
message is lost, that only the log's own expired archives are deleted,
and that the oldest archives are
deleted beyond the cap.
- `TestOutFileRotation` (5 cases): covers `rotate_out_file` with the
default, custom, zero, and invalid limits.
- helm-unittest cases for the rendered retention defaults and overrides.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 <[email protected]>
---
bin/common.sh.template | 37 +++
bin/gravitino-iceberg-rest-server.sh.template | 2 +
bin/gravitino-lance-rest-server.sh.template | 2 +
bin/gravitino.sh.template | 2 +
conf/gravitino-env.sh.template | 1 +
...avitino-iceberg-rest-log4j2.properties.template | 26 ++-
...gravitino-lance-rest-log4j2.properties.template | 26 ++-
conf/log4j2.properties.template | 63 ++++--
.../gravitino-iceberg-rest-log4j2.properties | 27 ++-
.../tests/configmap_test.yaml | 55 +++++
.../gravitino-iceberg-rest-server/values.yaml | 18 +-
.../gravitino-lance-rest-log4j2.properties | 27 ++-
.../tests/configmap_test.yaml | 60 +++++
dev/charts/gravitino-lance-rest-server/values.yaml | 5 +
.../gravitino/resources/config/log4j2.properties | 27 ++-
dev/charts/gravitino/tests/configmap_test.yaml | 55 +++++
dev/charts/gravitino/values.yaml | 18 +-
docs/gravitino-server-config.md | 18 +-
docs/how-to-install.md | 25 ++
.../server/TestLog4j2RollingConfiguration.java | 251 +++++++++++++++++++++
.../gravitino/server/TestOutFileRotation.java | 153 +++++++++++++
21 files changed, 830 insertions(+), 68 deletions(-)
diff --git a/bin/common.sh.template b/bin/common.sh.template
index b7642b3d3a..27d6af66b1 100644
--- a/bin/common.sh.template
+++ b/bin/common.sh.template
@@ -107,6 +107,43 @@ function addDirToClasspath(){
fi
}
+# Rotate a launch script's .out file before the process starts, so the file
does not grow across
+# restarts. The previous files are kept as <file>.1 (newest) to <file>.N
(oldest), where N is
+# GRAVITINO_OUT_FILE_KEEP (default 5). Set it to 0 to keep no previous file.
+function rotate_out_file() {
+ local out_file="$1"
+ local keep="${GRAVITINO_OUT_FILE_KEEP:-5}"
+ local rotated index
+
+ if ! [[ "${keep}" =~ ^[0-9]+$ ]]; then
+ echo "Invalid GRAVITINO_OUT_FILE_KEEP '${keep}', use 5 instead"
+ keep=5
+ fi
+
+ # Remove files beyond the limit, including those left by a larger earlier
limit.
+ for rotated in "${out_file}".*; do
+ index="${rotated##*.}"
+ if [[ -f "${rotated}" && "${index}" =~ ^[0-9]+$ ]] && (( 10#${index} >=
keep )); then
+ rm -f "${rotated}"
+ fi
+ done
+
+ if [[ ! -f "${out_file}" ]]; then
+ return 0
+ fi
+ if (( keep == 0 )); then
+ rm -f "${out_file}"
+ return 0
+ fi
+
+ for (( index = keep - 1; index >= 1; index-- )); do
+ if [[ -f "${out_file}.${index}" ]]; then
+ mv -f "${out_file}.${index}" "${out_file}.$((index + 1))"
+ fi
+ done
+ mv -f "${out_file}" "${out_file}.1"
+}
+
if [[ -z "${GRAVITINO_MEM}" ]]; then
export GRAVITINO_MEM="-Xms1024m -Xmx1024m -XX:MaxMetaspaceSize=512m"
fi
diff --git a/bin/gravitino-iceberg-rest-server.sh.template
b/bin/gravitino-iceberg-rest-server.sh.template
index 2f375a462b..f486378528 100755
--- a/bin/gravitino-iceberg-rest-server.sh.template
+++ b/bin/gravitino-iceberg-rest-server.sh.template
@@ -113,6 +113,8 @@ function start() {
mkdir -p "${GRAVITINO_LOG_DIR}"
fi
+ rotate_out_file "${GRAVITINO_OUTFILE}"
+
nohup ${JAVA_RUNNER} ${JAVA_OPTS} ${GRAVITINO_DEBUG_OPTS} -cp
${GRAVITINO_CLASSPATH} ${GRAVITINO_SERVER_NAME} >> "${GRAVITINO_OUTFILE}" 2>&1 &
pid=$!
diff --git a/bin/gravitino-lance-rest-server.sh.template
b/bin/gravitino-lance-rest-server.sh.template
index c8812f0860..b8537873cf 100644
--- a/bin/gravitino-lance-rest-server.sh.template
+++ b/bin/gravitino-lance-rest-server.sh.template
@@ -113,6 +113,8 @@ function start() {
mkdir -p "${GRAVITINO_LOG_DIR}"
fi
+ rotate_out_file "${GRAVITINO_OUTFILE}"
+
nohup ${JAVA_RUNNER} ${JAVA_OPTS} ${GRAVITINO_DEBUG_OPTS} -cp
${GRAVITINO_CLASSPATH} ${GRAVITINO_SERVER_NAME} >> "${GRAVITINO_OUTFILE}" 2>&1 &
pid=$!
diff --git a/bin/gravitino.sh.template b/bin/gravitino.sh.template
index 051de1fbc5..504755b868 100755
--- a/bin/gravitino.sh.template
+++ b/bin/gravitino.sh.template
@@ -113,6 +113,8 @@ function start() {
mkdir -p "${GRAVITINO_LOG_DIR}"
fi
+ rotate_out_file "${GRAVITINO_OUTFILE}"
+
nohup ${JAVA_RUNNER} ${JAVA_OPTS} ${GRAVITINO_DEBUG_OPTS} -cp
${GRAVITINO_CLASSPATH} ${GRAVITINO_SERVER_NAME} >> "${GRAVITINO_OUTFILE}" 2>&1 &
pid=$!
diff --git a/conf/gravitino-env.sh.template b/conf/gravitino-env.sh.template
index b80c398a33..8944622681 100644
--- a/conf/gravitino-env.sh.template
+++ b/conf/gravitino-env.sh.template
@@ -31,6 +31,7 @@ GRAVITINO_VERSION=GRAVITINO_VERSION_PLACEHOLDER
# export GRAVITINO_MEM # JVM memory options for the Gravitino server,
Iceberg REST server, and Lance REST server.
# # Default: -Xms1024m -Xmx1024m
-XX:MaxMetaspaceSize=512m
# # Appended to JAVA_OPTS by launch scripts; set
GRAVITINO_MEM to change heap/metaspace sizes.
+# export GRAVITINO_OUT_FILE_KEEP # Number of previous .out files kept when a
server starts. Default: 5.
# UI selection behaviour:
# - Default to the v2 UI when GRAVITINO_USE_WEB_V2 is unset or empty.
diff --git a/conf/gravitino-iceberg-rest-log4j2.properties.template
b/conf/gravitino-iceberg-rest-log4j2.properties.template
index 696d5a8878..0f5a14d68d 100644
--- a/conf/gravitino-iceberg-rest-log4j2.properties.template
+++ b/conf/gravitino-iceberg-rest-log4j2.properties.template
@@ -23,29 +23,41 @@ status = warn
property.basePath = ${sys:gravitino.iceberg-rest-server.log.path}
property.serverName = ${sys:gravitino.iceberg-rest-server.name}
+# Log retention. Rolled archives are kept for logMaxAge. Their total size is
also capped: when the
+# cap is exceeded, the oldest archives are deleted first. The cap does not
include the file that is
+# currently being written.
+property.logMaxAge = 30d
+property.serverLogMaxTotalSize = 2GB
+
# RollingFileAppender name, pattern, path and rollover policy
appender.rolling.type = RollingFile
appender.rolling.name = fileLogger
appender.rolling.fileName = ${basePath}/${serverName}.log
-appender.rolling.filePattern = ${basePath}/${serverName}_%d{yyyyMMdd}.log.gz
+appender.rolling.filePattern = ${basePath}/${serverName}_%d{yyyyMMdd}.%i.log.gz
appender.rolling.layout.type = PatternLayout
appender.rolling.layout.pattern = %d{yyyy-MM-dd HH:mm:ss.SSS} %level [%t] [%l]
- %msg%n
appender.rolling.policies.type = Policies
# RollingFileAppender rotation policy
appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
-appender.rolling.policies.size.size = 10MB
+appender.rolling.policies.size.size = 100MB
appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling.policies.time.interval = 1
appender.rolling.policies.time.modulate = true
appender.rolling.strategy.type = DefaultRolloverStrategy
+# Keep every archive of a day. Archives are removed only by the Delete action
below.
+appender.rolling.strategy.fileIndex = nomax
appender.rolling.strategy.delete.type = Delete
appender.rolling.strategy.delete.basePath = ${basePath}
-appender.rolling.strategy.delete.maxDepth = 10
-appender.rolling.strategy.delete.ifLastModified.type = IfLastModified
-
-# Delete all files older than 30 days
-appender.rolling.strategy.delete.ifLastModified.age = 30d
+appender.rolling.strategy.delete.maxDepth = 1
+appender.rolling.strategy.delete.ifFileName.type = IfFileName
+appender.rolling.strategy.delete.ifFileName.glob = ${serverName}_*.log.gz
+# Delete archives older than logMaxAge, and the oldest archives beyond
serverLogMaxTotalSize
+appender.rolling.strategy.delete.ifFileName.ifAny.type = IfAny
+appender.rolling.strategy.delete.ifFileName.ifAny.ifAccumulatedFileSize.type =
IfAccumulatedFileSize
+appender.rolling.strategy.delete.ifFileName.ifAny.ifAccumulatedFileSize.exceeds
= ${serverLogMaxTotalSize}
+appender.rolling.strategy.delete.ifFileName.ifAny.ifLastModified.type =
IfLastModified
+appender.rolling.strategy.delete.ifFileName.ifAny.ifLastModified.age =
${logMaxAge}
# Configure root logger
rootLogger.level = info
diff --git a/conf/gravitino-lance-rest-log4j2.properties.template
b/conf/gravitino-lance-rest-log4j2.properties.template
index bb8209d19f..97de49c701 100644
--- a/conf/gravitino-lance-rest-log4j2.properties.template
+++ b/conf/gravitino-lance-rest-log4j2.properties.template
@@ -23,29 +23,41 @@ status = warn
property.basePath = ${sys:gravitino.lance-rest-server.log.path}
property.serverName = ${sys:gravitino.lance-rest-server.name}
+# Log retention. Rolled archives are kept for logMaxAge. Their total size is
also capped: when the
+# cap is exceeded, the oldest archives are deleted first. The cap does not
include the file that is
+# currently being written.
+property.logMaxAge = 30d
+property.serverLogMaxTotalSize = 1GB
+
# RollingFileAppender name, pattern, path and rollover policy
appender.rolling.type = RollingFile
appender.rolling.name = fileLogger
appender.rolling.fileName = ${basePath}/${serverName}.log
-appender.rolling.filePattern = ${basePath}/${serverName}_%d{yyyyMMdd}.log.gz
+appender.rolling.filePattern = ${basePath}/${serverName}_%d{yyyyMMdd}.%i.log.gz
appender.rolling.layout.type = PatternLayout
appender.rolling.layout.pattern = %d{yyyy-MM-dd HH:mm:ss.SSS} %level [%t] [%l]
- %msg%n
appender.rolling.policies.type = Policies
# RollingFileAppender rotation policy
appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
-appender.rolling.policies.size.size = 10MB
+appender.rolling.policies.size.size = 100MB
appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling.policies.time.interval = 1
appender.rolling.policies.time.modulate = true
appender.rolling.strategy.type = DefaultRolloverStrategy
+# Keep every archive of a day. Archives are removed only by the Delete action
below.
+appender.rolling.strategy.fileIndex = nomax
appender.rolling.strategy.delete.type = Delete
appender.rolling.strategy.delete.basePath = ${basePath}
-appender.rolling.strategy.delete.maxDepth = 10
-appender.rolling.strategy.delete.ifLastModified.type = IfLastModified
-
-# Delete all files older than 30 days
-appender.rolling.strategy.delete.ifLastModified.age = 30d
+appender.rolling.strategy.delete.maxDepth = 1
+appender.rolling.strategy.delete.ifFileName.type = IfFileName
+appender.rolling.strategy.delete.ifFileName.glob = ${serverName}_*.log.gz
+# Delete archives older than logMaxAge, and the oldest archives beyond
serverLogMaxTotalSize
+appender.rolling.strategy.delete.ifFileName.ifAny.type = IfAny
+appender.rolling.strategy.delete.ifFileName.ifAny.ifAccumulatedFileSize.type =
IfAccumulatedFileSize
+appender.rolling.strategy.delete.ifFileName.ifAny.ifAccumulatedFileSize.exceeds
= ${serverLogMaxTotalSize}
+appender.rolling.strategy.delete.ifFileName.ifAny.ifLastModified.type =
IfLastModified
+appender.rolling.strategy.delete.ifFileName.ifAny.ifLastModified.age =
${logMaxAge}
# Configure root logger
rootLogger.level = info
diff --git a/conf/log4j2.properties.template b/conf/log4j2.properties.template
index 1952d6ae06..dcb290b998 100644
--- a/conf/log4j2.properties.template
+++ b/conf/log4j2.properties.template
@@ -23,49 +23,71 @@ status = warn
property.basePath = ${sys:gravitino.log.path}
property.serverName = ${sys:gravitino.server.name}
+# Log retention. Every log keeps its rolled archives for logMaxAge. The
archives of each log are
+# also capped in total size: when a cap is exceeded, the oldest archives of
that log are deleted
+# first. A cap does not include the file that is currently being written.
+property.logMaxAge = 30d
+property.serverLogMaxTotalSize = 2GB
+property.auditLogMaxTotalSize = 10GB
+property.lineageLogMaxTotalSize = 1GB
+
# RollingFileAppender name, pattern, path and rollover policy
appender.rolling.type = RollingFile
appender.rolling.name = fileLogger
appender.rolling.fileName = ${basePath}/${serverName}.log
-appender.rolling.filePattern = ${basePath}/${serverName}_%d{yyyyMMdd}.log.gz
+appender.rolling.filePattern = ${basePath}/${serverName}_%d{yyyyMMdd}.%i.log.gz
appender.rolling.layout.type = PatternLayout
appender.rolling.layout.pattern = %d{yyyy-MM-dd HH:mm:ss.SSS} %level [%t] [%l]
- %msg%n
appender.rolling.policies.type = Policies
# RollingFileAppender rotation policy
appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
-appender.rolling.policies.size.size = 10MB
+appender.rolling.policies.size.size = 100MB
appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling.policies.time.interval = 1
appender.rolling.policies.time.modulate = true
appender.rolling.strategy.type = DefaultRolloverStrategy
+# Keep every archive of a day. Archives are removed only by the Delete action
below.
+appender.rolling.strategy.fileIndex = nomax
appender.rolling.strategy.delete.type = Delete
appender.rolling.strategy.delete.basePath = ${basePath}
-appender.rolling.strategy.delete.maxDepth = 10
-appender.rolling.strategy.delete.ifLastModified.type = IfLastModified
-
-# Delete all files older than 30 days
-appender.rolling.strategy.delete.ifLastModified.age = 30d
+appender.rolling.strategy.delete.maxDepth = 1
+appender.rolling.strategy.delete.ifFileName.type = IfFileName
+appender.rolling.strategy.delete.ifFileName.glob = ${serverName}_*.log.gz
+# Delete archives older than logMaxAge, and the oldest archives beyond
serverLogMaxTotalSize
+appender.rolling.strategy.delete.ifFileName.ifAny.type = IfAny
+appender.rolling.strategy.delete.ifFileName.ifAny.ifAccumulatedFileSize.type =
IfAccumulatedFileSize
+appender.rolling.strategy.delete.ifFileName.ifAny.ifAccumulatedFileSize.exceeds
= ${serverLogMaxTotalSize}
+appender.rolling.strategy.delete.ifFileName.ifAny.ifLastModified.type =
IfLastModified
+appender.rolling.strategy.delete.ifFileName.ifAny.ifLastModified.age =
${logMaxAge}
## use separate file for lineage log
appender.lineage_file.type=RollingFile
appender.lineage_file.name=lineage_file
appender.lineage_file.fileName=${basePath}/gravitino_lineage.log
-appender.lineage_file.filePattern=${basePath}/gravitino_lineage_%d{yyyyMMdd}.log.gz
+appender.lineage_file.filePattern=${basePath}/gravitino_lineage_%d{yyyyMMdd}.%i.log.gz
appender.lineage_file.layout.type=PatternLayout
appender.lineage_file.layout.pattern=[%d{yyyy-MM-dd HH:mm:ss}] %m%n
appender.lineage_file.policies.type = Policies
-
+appender.lineage_file.policies.size.type = SizeBasedTriggeringPolicy
+appender.lineage_file.policies.size.size = 100MB
appender.lineage_file.policies.time.type = TimeBasedTriggeringPolicy
appender.lineage_file.policies.time.interval = 1
appender.lineage_file.policies.time.modulate = true
appender.lineage_file.strategy.type = DefaultRolloverStrategy
+appender.lineage_file.strategy.fileIndex = nomax
appender.lineage_file.strategy.delete.type = Delete
appender.lineage_file.strategy.delete.basePath = ${basePath}
-appender.lineage_file.strategy.delete.maxDepth = 10
-appender.lineage_file.strategy.delete.ifLastModified.type = IfLastModified
-appender.lineage_file.strategy.delete.ifLastModified.age = 30d
+appender.lineage_file.strategy.delete.maxDepth = 1
+appender.lineage_file.strategy.delete.ifFileName.type = IfFileName
+appender.lineage_file.strategy.delete.ifFileName.glob =
gravitino_lineage_*.log.gz
+# Delete archives older than logMaxAge, and the oldest archives beyond
lineageLogMaxTotalSize
+appender.lineage_file.strategy.delete.ifFileName.ifAny.type = IfAny
+appender.lineage_file.strategy.delete.ifFileName.ifAny.ifAccumulatedFileSize.type
= IfAccumulatedFileSize
+appender.lineage_file.strategy.delete.ifFileName.ifAny.ifAccumulatedFileSize.exceeds
= ${lineageLogMaxTotalSize}
+appender.lineage_file.strategy.delete.ifFileName.ifAny.ifLastModified.type =
IfLastModified
+appender.lineage_file.strategy.delete.ifFileName.ifAny.ifLastModified.age =
${logMaxAge}
logger.lineage.name =
org.apache.gravitino.lineage.sink.LineageLogSink$LineageLogger
logger.lineage.level = info
@@ -86,15 +108,18 @@ appender.audit_file.policies.time.type =
TimeBasedTriggeringPolicy
appender.audit_file.policies.time.interval = 1
appender.audit_file.policies.time.modulate = true
appender.audit_file.strategy.type = DefaultRolloverStrategy
+appender.audit_file.strategy.fileIndex = nomax
appender.audit_file.strategy.delete.type = Delete
appender.audit_file.strategy.delete.basePath = ${basePath}
-appender.audit_file.strategy.delete.maxDepth = 10
-appender.audit_file.strategy.delete.ifAll.type = IfAll
-appender.audit_file.strategy.delete.ifAll.ifFileName.type = IfFileName
-appender.audit_file.strategy.delete.ifAll.ifFileName.regex =
gravitino_audit.*\.log\.gz
-# Delete all audit log files older than 30 days
-appender.audit_file.strategy.delete.ifAll.ifLastModified.type = IfLastModified
-appender.audit_file.strategy.delete.ifAll.ifLastModified.age = 30d
+appender.audit_file.strategy.delete.maxDepth = 1
+appender.audit_file.strategy.delete.ifFileName.type = IfFileName
+appender.audit_file.strategy.delete.ifFileName.glob = gravitino_audit_*.log.gz
+# Delete audit archives older than logMaxAge, and the oldest archives beyond
auditLogMaxTotalSize
+appender.audit_file.strategy.delete.ifFileName.ifAny.type = IfAny
+appender.audit_file.strategy.delete.ifFileName.ifAny.ifAccumulatedFileSize.type
= IfAccumulatedFileSize
+appender.audit_file.strategy.delete.ifFileName.ifAny.ifAccumulatedFileSize.exceeds
= ${auditLogMaxTotalSize}
+appender.audit_file.strategy.delete.ifFileName.ifAny.ifLastModified.type =
IfLastModified
+appender.audit_file.strategy.delete.ifFileName.ifAny.ifLastModified.age =
${logMaxAge}
logger.audit.name = gravitino.audit
logger.audit.level = info
diff --git
a/dev/charts/gravitino-iceberg-rest-server/resources/gravitino-iceberg-rest-log4j2.properties
b/dev/charts/gravitino-iceberg-rest-server/resources/gravitino-iceberg-rest-log4j2.properties
index 5a49bc567b..42623c22f6 100755
---
a/dev/charts/gravitino-iceberg-rest-server/resources/gravitino-iceberg-rest-log4j2.properties
+++
b/dev/charts/gravitino-iceberg-rest-server/resources/gravitino-iceberg-rest-log4j2.properties
@@ -23,29 +23,42 @@ status = {{ .Values.log4j2Properties.status | default
"warn" }}
property.basePath = {{ .Values.log4j2Properties.basePath | default
"${sys:gravitino.iceberg-rest-server.log.path}" }}
property.serverName = {{ .Values.log4j2Properties.serverName | default
"${sys:gravitino.iceberg-rest-server.name}" }}
+# Log retention. Rolled archives are kept for logMaxAge. Their total size is
also capped: when the
+# cap is exceeded, the oldest archives are deleted first. The cap does not
include the file that is
+# currently being written.
+property.logMaxAge = {{ .Values.log4j2Properties.logMaxAge | default "30d" }}
+property.serverLogMaxTotalSize = {{
.Values.log4j2Properties.serverLogMaxTotalSize | default "2GB" }}
+
# RollingFileAppender name, pattern, path and rollover policy
appender.rolling.type = {{ .Values.log4j2Properties.rollingAppenderType |
default "RollingFile" }}
appender.rolling.name = {{ .Values.log4j2Properties.rollingAppenderName |
default "fileLogger" }}
appender.rolling.fileName = {{
.Values.log4j2Properties.rollingAppenderFileName | default
"${basePath}/${serverName}.log" }}
-appender.rolling.filePattern = {{
.Values.log4j2Properties.rollingAppenderFilePattern | default
"${basePath}/${serverName}_%d{yyyyMMdd}.log.gz" }}
+appender.rolling.filePattern = {{
.Values.log4j2Properties.rollingAppenderFilePattern | default
"${basePath}/${serverName}_%d{yyyyMMdd}.%i.log.gz" }}
appender.rolling.layout.type = {{
.Values.log4j2Properties.rollingAppenderLayoutType | default "PatternLayout" }}
appender.rolling.layout.pattern = {{
.Values.log4j2Properties.rollingAppenderLayoutPattern | default "%d{yyyy-MM-dd
HH:mm:ss.SSS} %level [%t] [%l] - %msg%n" }}
appender.rolling.policies.type = {{
.Values.log4j2Properties.rollingAppenderPoliciesType | default "Policies" }}
# RollingFileAppender rotation policy
appender.rolling.policies.size.type = {{
.Values.log4j2Properties.rollingAppenderPoliciesSizeType | default
"SizeBasedTriggeringPolicy" }}
-appender.rolling.policies.size.size = {{
.Values.log4j2Properties.rollingAppenderPoliciesSizeSize | default "10MB" }}
+appender.rolling.policies.size.size = {{
.Values.log4j2Properties.rollingAppenderPoliciesSizeSize | default "100MB" }}
appender.rolling.policies.time.type = {{
.Values.log4j2Properties.rollingAppenderPoliciesTimeType | default
"TimeBasedTriggeringPolicy" }}
appender.rolling.policies.time.interval = {{
.Values.log4j2Properties.rollingAppenderPoliciesTimeInterval | default 1 }}
appender.rolling.policies.time.modulate = {{
.Values.log4j2Properties.rollingAppenderPoliciesTimeModulate | default true }}
appender.rolling.strategy.type = {{
.Values.log4j2Properties.rollingAppenderStrategyType | default
"DefaultRolloverStrategy" }}
+# Keep every archive of a day. Archives are removed only by the Delete action
below.
+appender.rolling.strategy.fileIndex = {{
.Values.log4j2Properties.rollingAppenderStrategyFileIndex | default "nomax" }}
appender.rolling.strategy.delete.type = {{
.Values.log4j2Properties.rollingAppenderStrategyDeleteType | default "Delete" }}
appender.rolling.strategy.delete.basePath = {{
.Values.log4j2Properties.rollingAppenderStrategyDeleteBasePath | default
"${basePath}" }}
-appender.rolling.strategy.delete.maxDepth = {{
.Values.log4j2Properties.rollingAppenderStrategyDeleteMaxDepth | default 10 }}
-appender.rolling.strategy.delete.ifLastModified.type = {{
.Values.log4j2Properties.rollingAppenderStrategyDeleteIfLastModifiedType |
default "IfLastModified" }}
-
-# Delete all files older than 30 days
-appender.rolling.strategy.delete.ifLastModified.age = {{
.Values.log4j2Properties.rollingAppenderStrategyDeleteIfLastModifiedAge |
default "30d" }}
+appender.rolling.strategy.delete.maxDepth = {{
.Values.log4j2Properties.rollingAppenderStrategyDeleteMaxDepth | default 1 }}
+appender.rolling.strategy.delete.ifFileName.type = IfFileName
+# Must match the archive names of rollingAppenderFilePattern, or archives are
never deleted.
+appender.rolling.strategy.delete.ifFileName.glob = {{
.Values.log4j2Properties.rollingAppenderStrategyDeleteIfFileNameGlob | default
"${serverName}_*.log.gz" }}
+# Delete archives older than logMaxAge, and the oldest archives beyond
serverLogMaxTotalSize
+appender.rolling.strategy.delete.ifFileName.ifAny.type = IfAny
+appender.rolling.strategy.delete.ifFileName.ifAny.ifAccumulatedFileSize.type =
IfAccumulatedFileSize
+appender.rolling.strategy.delete.ifFileName.ifAny.ifAccumulatedFileSize.exceeds
= ${serverLogMaxTotalSize}
+appender.rolling.strategy.delete.ifFileName.ifAny.ifLastModified.type = {{
.Values.log4j2Properties.rollingAppenderStrategyDeleteIfLastModifiedType |
default "IfLastModified" }}
+appender.rolling.strategy.delete.ifFileName.ifAny.ifLastModified.age = {{
.Values.log4j2Properties.rollingAppenderStrategyDeleteIfLastModifiedAge |
default "${logMaxAge}" }}
# Configure root logger
rootLogger.level = {{ .Values.log4j2Properties.rootLoggerLevel | default
"info" }}
diff --git a/dev/charts/gravitino-iceberg-rest-server/tests/configmap_test.yaml
b/dev/charts/gravitino-iceberg-rest-server/tests/configmap_test.yaml
index c76bf301d5..29d819898c 100644
--- a/dev/charts/gravitino-iceberg-rest-server/tests/configmap_test.yaml
+++ b/dev/charts/gravitino-iceberg-rest-server/tests/configmap_test.yaml
@@ -154,3 +154,58 @@ tests:
- matchRegex:
path: data["hdfs-site.xml"]
pattern: "<value>3</value>"
+
+ - it: renders log rotation and retention defaults
+ release:
+ name: iceberg-rest
+ asserts:
+ - matchRegex:
+ path: data["gravitino-iceberg-rest-log4j2.properties"]
+ pattern: "property\\.logMaxAge = 30d"
+ - matchRegex:
+ path: data["gravitino-iceberg-rest-log4j2.properties"]
+ pattern: "property\\.serverLogMaxTotalSize = 2GB"
+ - matchRegex:
+ path: data["gravitino-iceberg-rest-log4j2.properties"]
+ pattern: "appender\\.rolling\\.filePattern =
\\$\\{basePath\\}/\\$\\{serverName\\}_%d\\{yyyyMMdd\\}\\.%i\\.log\\.gz"
+ - matchRegex:
+ path: data["gravitino-iceberg-rest-log4j2.properties"]
+ pattern: "appender\\.rolling\\.policies\\.size\\.size = 100MB"
+ - matchRegex:
+ path: data["gravitino-iceberg-rest-log4j2.properties"]
+ pattern: "appender\\.rolling\\.strategy\\.fileIndex = nomax"
+ - matchRegex:
+ path: data["gravitino-iceberg-rest-log4j2.properties"]
+ pattern: "appender\\.rolling\\.strategy\\.delete\\.maxDepth = 1"
+ - matchRegex:
+ path: data["gravitino-iceberg-rest-log4j2.properties"]
+ pattern: "appender\\.rolling\\.strategy\\.delete\\.ifFileName\\.glob
= \\$\\{serverName\\}_\\*\\.log\\.gz"
+ - matchRegex:
+ path: data["gravitino-iceberg-rest-log4j2.properties"]
+ pattern:
"appender\\.rolling\\.strategy\\.delete\\.ifFileName\\.ifAny\\.ifAccumulatedFileSize\\.exceeds
= \\$\\{serverLogMaxTotalSize\\}"
+ - matchRegex:
+ path: data["gravitino-iceberg-rest-log4j2.properties"]
+ pattern:
"appender\\.rolling\\.strategy\\.delete\\.ifFileName\\.ifAny\\.ifLastModified\\.age
= \\$\\{logMaxAge\\}"
+
+ - it: renders customized log retention
+ release:
+ name: iceberg-rest
+ set:
+ log4j2Properties:
+ logMaxAge: 7d
+ serverLogMaxTotalSize: 500MB
+ rollingAppenderPoliciesSizeSize: 20MB
+ rollingAppenderStrategyDeleteIfFileNameGlob: "app_*.log.gz"
+ asserts:
+ - matchRegex:
+ path: data["gravitino-iceberg-rest-log4j2.properties"]
+ pattern: "property\\.logMaxAge = 7d"
+ - matchRegex:
+ path: data["gravitino-iceberg-rest-log4j2.properties"]
+ pattern: "property\\.serverLogMaxTotalSize = 500MB"
+ - matchRegex:
+ path: data["gravitino-iceberg-rest-log4j2.properties"]
+ pattern: "appender\\.rolling\\.policies\\.size\\.size = 20MB"
+ - matchRegex:
+ path: data["gravitino-iceberg-rest-log4j2.properties"]
+ pattern: "appender\\.rolling\\.strategy\\.delete\\.ifFileName\\.glob
= app_\\*\\.log\\.gz"
diff --git a/dev/charts/gravitino-iceberg-rest-server/values.yaml
b/dev/charts/gravitino-iceberg-rest-server/values.yaml
index ed7b9cc8e2..9550b1f60b 100644
--- a/dev/charts/gravitino-iceberg-rest-server/values.yaml
+++ b/dev/charts/gravitino-iceberg-rest-server/values.yaml
@@ -160,29 +160,37 @@ log4j2Properties: {}
# basePath: "${sys:gravitino.log.path}"
# serverName: "${sys:gravitino.server.name}"
+ ## Log retention. Rolled archives are kept for logMaxAge. Their total size
is also capped: when the
+ ## cap is exceeded, the oldest archives are deleted first.
+ # logMaxAge: 30d
+ # serverLogMaxTotalSize: 2GB
+
## RollingFileAppender name, pattern, path and rollover policy
# rollingAppenderType: RollingFile
# rollingAppenderName: fileLogger
# rollingAppenderFileName: "${basePath}/${serverName}.log"
- # rollingAppenderFilePattern: "${basePath}/${serverName}_%d{yyyyMMdd}.log.gz"
+ # rollingAppenderFilePattern:
"${basePath}/${serverName}_%d{yyyyMMdd}.%i.log.gz"
+ ## When you change the archive name, set
rollingAppenderStrategyDeleteIfFileNameGlob to match it.
# rollingAppenderLayoutType: PatternLayout
# rollingAppenderLayoutPattern: "%d{yyyy-MM-dd HH:mm:ss.SSS} %level [%t]
[%l] - %msg%n"
# rollingAppenderPoliciesType: Policies
## RollingFileAppender rotation policy
# rollingAppenderPoliciesSizeType: SizeBasedTriggeringPolicy
- # rollingAppenderPoliciesSizeSize: 10MB
+ # rollingAppenderPoliciesSizeSize: 100MB
# rollingAppenderPoliciesTimeType: TimeBasedTriggeringPolicy
# rollingAppenderPoliciesTimeInterval: 1
# rollingAppenderPoliciesTimeModulate: true
# rollingAppenderStrategyType: DefaultRolloverStrategy
+ # rollingAppenderStrategyFileIndex: nomax
# rollingAppenderStrategyDeleteType: Delete
# rollingAppenderStrategyDeleteBasePath: "${basePath}"
- # rollingAppenderStrategyDeleteMaxDepth: 10
+ # rollingAppenderStrategyDeleteMaxDepth: 1
+ # rollingAppenderStrategyDeleteIfFileNameGlob: "${serverName}_*.log.gz"
# rollingAppenderStrategyDeleteIfLastModifiedType: IfLastModified
- ## Delete all files older than 30 days
- # rollingAppenderStrategyDeleteIfLastModifiedAge: 30d
+ ## Delete archives older than this age. Defaults to logMaxAge.
+ # rollingAppenderStrategyDeleteIfLastModifiedAge: "${logMaxAge}"
## Lineage log appender configurations
# lineageFileType: RollingFile
diff --git
a/dev/charts/gravitino-lance-rest-server/resources/gravitino-lance-rest-log4j2.properties
b/dev/charts/gravitino-lance-rest-server/resources/gravitino-lance-rest-log4j2.properties
index 6bbb90edfe..68a0be5768 100644
---
a/dev/charts/gravitino-lance-rest-server/resources/gravitino-lance-rest-log4j2.properties
+++
b/dev/charts/gravitino-lance-rest-server/resources/gravitino-lance-rest-log4j2.properties
@@ -23,29 +23,42 @@ status = {{ .Values.log4j2Properties.status | default
"warn" }}
property.basePath = {{ .Values.log4j2Properties.basePath | default
"${sys:gravitino.lance-rest-server.log.path}" }}
property.serverName = {{ .Values.log4j2Properties.serverName | default
"${sys:gravitino.lance-rest-server.name}" }}
+# Log retention. Rolled archives are kept for logMaxAge. Their total size is
also capped: when the
+# cap is exceeded, the oldest archives are deleted first. The cap does not
include the file that is
+# currently being written.
+property.logMaxAge = {{ .Values.log4j2Properties.logMaxAge | default "30d" }}
+property.serverLogMaxTotalSize = {{
.Values.log4j2Properties.serverLogMaxTotalSize | default "1GB" }}
+
# RollingFileAppender name, pattern, path and rollover policy
appender.rolling.type = {{ .Values.log4j2Properties.rollingAppenderType |
default "RollingFile" }}
appender.rolling.name = {{ .Values.log4j2Properties.rollingAppenderName |
default "fileLogger" }}
appender.rolling.fileName = {{
.Values.log4j2Properties.rollingAppenderFileName | default
"${basePath}/${serverName}.log" }}
-appender.rolling.filePattern = {{
.Values.log4j2Properties.rollingAppenderFilePattern | default
"${basePath}/${serverName}_%d{yyyyMMdd}.log.gz" }}
+appender.rolling.filePattern = {{
.Values.log4j2Properties.rollingAppenderFilePattern | default
"${basePath}/${serverName}_%d{yyyyMMdd}.%i.log.gz" }}
appender.rolling.layout.type = {{
.Values.log4j2Properties.rollingAppenderLayoutType | default "PatternLayout" }}
appender.rolling.layout.pattern = {{
.Values.log4j2Properties.rollingAppenderLayoutPattern | default "%d{yyyy-MM-dd
HH:mm:ss.SSS} %level [%t] [%l] - %msg%n" }}
appender.rolling.policies.type = {{
.Values.log4j2Properties.rollingAppenderPoliciesType | default "Policies" }}
# RollingFileAppender rotation policy
appender.rolling.policies.size.type = {{
.Values.log4j2Properties.rollingAppenderPoliciesSizeType | default
"SizeBasedTriggeringPolicy" }}
-appender.rolling.policies.size.size = {{
.Values.log4j2Properties.rollingAppenderPoliciesSizeSize | default "10MB" }}
+appender.rolling.policies.size.size = {{
.Values.log4j2Properties.rollingAppenderPoliciesSizeSize | default "100MB" }}
appender.rolling.policies.time.type = {{
.Values.log4j2Properties.rollingAppenderPoliciesTimeType | default
"TimeBasedTriggeringPolicy" }}
appender.rolling.policies.time.interval = {{
.Values.log4j2Properties.rollingAppenderPoliciesTimeInterval | default 1 }}
appender.rolling.policies.time.modulate = {{
.Values.log4j2Properties.rollingAppenderPoliciesTimeModulate | default true }}
appender.rolling.strategy.type = {{
.Values.log4j2Properties.rollingAppenderStrategyType | default
"DefaultRolloverStrategy" }}
+# Keep every archive of a day. Archives are removed only by the Delete action
below.
+appender.rolling.strategy.fileIndex = {{
.Values.log4j2Properties.rollingAppenderStrategyFileIndex | default "nomax" }}
appender.rolling.strategy.delete.type = {{
.Values.log4j2Properties.rollingAppenderStrategyDeleteType | default "Delete" }}
appender.rolling.strategy.delete.basePath = {{
.Values.log4j2Properties.rollingAppenderStrategyDeleteBasePath | default
"${basePath}" }}
-appender.rolling.strategy.delete.maxDepth = {{
.Values.log4j2Properties.rollingAppenderStrategyDeleteMaxDepth | default 10 }}
-appender.rolling.strategy.delete.ifLastModified.type = {{
.Values.log4j2Properties.rollingAppenderStrategyDeleteIfLastModifiedType |
default "IfLastModified" }}
-
-# Delete all files older than 30 days
-appender.rolling.strategy.delete.ifLastModified.age = {{
.Values.log4j2Properties.rollingAppenderStrategyDeleteIfLastModifiedAge |
default "30d" }}
+appender.rolling.strategy.delete.maxDepth = {{
.Values.log4j2Properties.rollingAppenderStrategyDeleteMaxDepth | default 1 }}
+appender.rolling.strategy.delete.ifFileName.type = IfFileName
+# Must match the archive names of rollingAppenderFilePattern, or archives are
never deleted.
+appender.rolling.strategy.delete.ifFileName.glob = {{
.Values.log4j2Properties.rollingAppenderStrategyDeleteIfFileNameGlob | default
"${serverName}_*.log.gz" }}
+# Delete archives older than logMaxAge, and the oldest archives beyond
serverLogMaxTotalSize
+appender.rolling.strategy.delete.ifFileName.ifAny.type = IfAny
+appender.rolling.strategy.delete.ifFileName.ifAny.ifAccumulatedFileSize.type =
IfAccumulatedFileSize
+appender.rolling.strategy.delete.ifFileName.ifAny.ifAccumulatedFileSize.exceeds
= ${serverLogMaxTotalSize}
+appender.rolling.strategy.delete.ifFileName.ifAny.ifLastModified.type = {{
.Values.log4j2Properties.rollingAppenderStrategyDeleteIfLastModifiedType |
default "IfLastModified" }}
+appender.rolling.strategy.delete.ifFileName.ifAny.ifLastModified.age = {{
.Values.log4j2Properties.rollingAppenderStrategyDeleteIfLastModifiedAge |
default "${logMaxAge}" }}
# Configure root logger
rootLogger.level = {{ .Values.log4j2Properties.rootLoggerLevel | default
"info" }}
diff --git a/dev/charts/gravitino-lance-rest-server/tests/configmap_test.yaml
b/dev/charts/gravitino-lance-rest-server/tests/configmap_test.yaml
index f03ba4ef67..031ecdb9d6 100644
--- a/dev/charts/gravitino-lance-rest-server/tests/configmap_test.yaml
+++ b/dev/charts/gravitino-lance-rest-server/tests/configmap_test.yaml
@@ -135,3 +135,63 @@ tests:
- notMatchRegex:
path: data["gravitino-lance-rest-server.conf"]
pattern: "gravitino\\.lance-rest\\.gravitino-metalake ="
+
+ - it: renders log rotation and retention defaults
+ release:
+ name: lance-rest
+ set:
+ lanceRest:
+ gravitinoMetalake: lrs_test
+ asserts:
+ - matchRegex:
+ path: data["gravitino-lance-rest-log4j2.properties"]
+ pattern: "property\\.logMaxAge = 30d"
+ - matchRegex:
+ path: data["gravitino-lance-rest-log4j2.properties"]
+ pattern: "property\\.serverLogMaxTotalSize = 1GB"
+ - matchRegex:
+ path: data["gravitino-lance-rest-log4j2.properties"]
+ pattern: "appender\\.rolling\\.filePattern =
\\$\\{basePath\\}/\\$\\{serverName\\}_%d\\{yyyyMMdd\\}\\.%i\\.log\\.gz"
+ - matchRegex:
+ path: data["gravitino-lance-rest-log4j2.properties"]
+ pattern: "appender\\.rolling\\.policies\\.size\\.size = 100MB"
+ - matchRegex:
+ path: data["gravitino-lance-rest-log4j2.properties"]
+ pattern: "appender\\.rolling\\.strategy\\.fileIndex = nomax"
+ - matchRegex:
+ path: data["gravitino-lance-rest-log4j2.properties"]
+ pattern: "appender\\.rolling\\.strategy\\.delete\\.maxDepth = 1"
+ - matchRegex:
+ path: data["gravitino-lance-rest-log4j2.properties"]
+ pattern: "appender\\.rolling\\.strategy\\.delete\\.ifFileName\\.glob
= \\$\\{serverName\\}_\\*\\.log\\.gz"
+ - matchRegex:
+ path: data["gravitino-lance-rest-log4j2.properties"]
+ pattern:
"appender\\.rolling\\.strategy\\.delete\\.ifFileName\\.ifAny\\.ifAccumulatedFileSize\\.exceeds
= \\$\\{serverLogMaxTotalSize\\}"
+ - matchRegex:
+ path: data["gravitino-lance-rest-log4j2.properties"]
+ pattern:
"appender\\.rolling\\.strategy\\.delete\\.ifFileName\\.ifAny\\.ifLastModified\\.age
= \\$\\{logMaxAge\\}"
+
+ - it: renders customized log retention
+ release:
+ name: lance-rest
+ set:
+ lanceRest:
+ gravitinoMetalake: lrs_test
+ log4j2Properties:
+ logMaxAge: 7d
+ serverLogMaxTotalSize: 500MB
+ rollingAppenderPoliciesSizeSize: 20MB
+ rollingAppenderStrategyDeleteIfFileNameGlob: "app_*.log.gz"
+ asserts:
+ - matchRegex:
+ path: data["gravitino-lance-rest-log4j2.properties"]
+ pattern: "property\\.logMaxAge = 7d"
+ - matchRegex:
+ path: data["gravitino-lance-rest-log4j2.properties"]
+ pattern: "property\\.serverLogMaxTotalSize = 500MB"
+ - matchRegex:
+ path: data["gravitino-lance-rest-log4j2.properties"]
+ pattern: "appender\\.rolling\\.policies\\.size\\.size = 20MB"
+ - matchRegex:
+ path: data["gravitino-lance-rest-log4j2.properties"]
+ pattern: "appender\\.rolling\\.strategy\\.delete\\.ifFileName\\.glob
= app_\\*\\.log\\.gz"
diff --git a/dev/charts/gravitino-lance-rest-server/values.yaml
b/dev/charts/gravitino-lance-rest-server/values.yaml
index 0252f1f22a..42bc2cf9d9 100644
--- a/dev/charts/gravitino-lance-rest-server/values.yaml
+++ b/dev/charts/gravitino-lance-rest-server/values.yaml
@@ -58,6 +58,11 @@ additionalConfigItems: {}
## Log4j2 configuration items
log4j2Properties:
status: warn
+ ## Log retention. Rolled archives are kept for logMaxAge. Their total size
is also capped: when the
+ ## cap is exceeded, the oldest archives are deleted first.
+ # logMaxAge: 30d
+ # serverLogMaxTotalSize: 1GB
+
additionalLog4j2Properties:
appender.console.type: Console
appender.console.name: consoleLogger
diff --git a/dev/charts/gravitino/resources/config/log4j2.properties
b/dev/charts/gravitino/resources/config/log4j2.properties
index e6101c9517..db645210fa 100644
--- a/dev/charts/gravitino/resources/config/log4j2.properties
+++ b/dev/charts/gravitino/resources/config/log4j2.properties
@@ -23,29 +23,42 @@ status = {{ .Values.log4j2Properties.status | default
"warn" }}
property.basePath = {{ .Values.log4j2Properties.basePath | default
"${sys:gravitino.log.path}" }}
property.serverName = {{ .Values.log4j2Properties.serverName | default
"${sys:gravitino.server.name}" }}
+# Log retention. Rolled archives are kept for logMaxAge. Their total size is
also capped: when the
+# cap is exceeded, the oldest archives are deleted first. The cap does not
include the file that is
+# currently being written.
+property.logMaxAge = {{ .Values.log4j2Properties.logMaxAge | default "30d" }}
+property.serverLogMaxTotalSize = {{
.Values.log4j2Properties.serverLogMaxTotalSize | default "2GB" }}
+
# RollingFileAppender name, pattern, path and rollover policy
appender.rolling.type = {{ .Values.log4j2Properties.rollingAppenderType |
default "RollingFile" }}
appender.rolling.name = {{ .Values.log4j2Properties.rollingAppenderName |
default "fileLogger" }}
appender.rolling.fileName = {{
.Values.log4j2Properties.rollingAppenderFileName | default
"${basePath}/${serverName}.log" }}
-appender.rolling.filePattern = {{
.Values.log4j2Properties.rollingAppenderFilePattern | default
"${basePath}/${serverName}_%d{yyyyMMdd}.log.gz" }}
+appender.rolling.filePattern = {{
.Values.log4j2Properties.rollingAppenderFilePattern | default
"${basePath}/${serverName}_%d{yyyyMMdd}.%i.log.gz" }}
appender.rolling.layout.type = {{
.Values.log4j2Properties.rollingAppenderLayoutType | default "PatternLayout" }}
appender.rolling.layout.pattern = {{
.Values.log4j2Properties.rollingAppenderLayoutPattern | default "%d{yyyy-MM-dd
HH:mm:ss.SSS} %level [%t] [%l] - %msg%n" }}
appender.rolling.policies.type = {{
.Values.log4j2Properties.rollingAppenderPoliciesType | default "Policies" }}
# RollingFileAppender rotation policy
appender.rolling.policies.size.type = {{
.Values.log4j2Properties.rollingAppenderPoliciesSizeType | default
"SizeBasedTriggeringPolicy" }}
-appender.rolling.policies.size.size = {{
.Values.log4j2Properties.rollingAppenderPoliciesSizeSize | default "10MB" }}
+appender.rolling.policies.size.size = {{
.Values.log4j2Properties.rollingAppenderPoliciesSizeSize | default "100MB" }}
appender.rolling.policies.time.type = {{
.Values.log4j2Properties.rollingAppenderPoliciesTimeType | default
"TimeBasedTriggeringPolicy" }}
appender.rolling.policies.time.interval = {{
.Values.log4j2Properties.rollingAppenderPoliciesTimeInterval | default 1 }}
appender.rolling.policies.time.modulate = {{
.Values.log4j2Properties.rollingAppenderPoliciesTimeModulate | default true }}
appender.rolling.strategy.type = {{
.Values.log4j2Properties.rollingAppenderStrategyType | default
"DefaultRolloverStrategy" }}
+# Keep every archive of a day. Archives are removed only by the Delete action
below.
+appender.rolling.strategy.fileIndex = {{
.Values.log4j2Properties.rollingAppenderStrategyFileIndex | default "nomax" }}
appender.rolling.strategy.delete.type = {{
.Values.log4j2Properties.rollingAppenderStrategyDeleteType | default "Delete" }}
appender.rolling.strategy.delete.basePath = {{
.Values.log4j2Properties.rollingAppenderStrategyDeleteBasePath | default
"${basePath}" }}
-appender.rolling.strategy.delete.maxDepth = {{
.Values.log4j2Properties.rollingAppenderStrategyDeleteMaxDepth | default 10 }}
-appender.rolling.strategy.delete.ifLastModified.type = {{
.Values.log4j2Properties.rollingAppenderStrategyDeleteIfLastModifiedType |
default "IfLastModified" }}
-
-# Delete all files older than 30 days
-appender.rolling.strategy.delete.ifLastModified.age = {{
.Values.log4j2Properties.rollingAppenderStrategyDeleteIfLastModifiedAge |
default "30d" }}
+appender.rolling.strategy.delete.maxDepth = {{
.Values.log4j2Properties.rollingAppenderStrategyDeleteMaxDepth | default 1 }}
+appender.rolling.strategy.delete.ifFileName.type = IfFileName
+# Must match the archive names of rollingAppenderFilePattern, or archives are
never deleted.
+appender.rolling.strategy.delete.ifFileName.glob = {{
.Values.log4j2Properties.rollingAppenderStrategyDeleteIfFileNameGlob | default
"${serverName}_*.log.gz" }}
+# Delete archives older than logMaxAge, and the oldest archives beyond
serverLogMaxTotalSize
+appender.rolling.strategy.delete.ifFileName.ifAny.type = IfAny
+appender.rolling.strategy.delete.ifFileName.ifAny.ifAccumulatedFileSize.type =
IfAccumulatedFileSize
+appender.rolling.strategy.delete.ifFileName.ifAny.ifAccumulatedFileSize.exceeds
= ${serverLogMaxTotalSize}
+appender.rolling.strategy.delete.ifFileName.ifAny.ifLastModified.type = {{
.Values.log4j2Properties.rollingAppenderStrategyDeleteIfLastModifiedType |
default "IfLastModified" }}
+appender.rolling.strategy.delete.ifFileName.ifAny.ifLastModified.age = {{
.Values.log4j2Properties.rollingAppenderStrategyDeleteIfLastModifiedAge |
default "${logMaxAge}" }}
# Suppress verbose JCasbin authorization decision logs by default
logger.jcasbin.name = org.casbin.jcasbin
diff --git a/dev/charts/gravitino/tests/configmap_test.yaml
b/dev/charts/gravitino/tests/configmap_test.yaml
index d0d9964d86..761c007847 100644
--- a/dev/charts/gravitino/tests/configmap_test.yaml
+++ b/dev/charts/gravitino/tests/configmap_test.yaml
@@ -336,3 +336,58 @@ tests:
- notMatchRegex:
path: data["gravitino.conf"]
pattern: "gravitino\\.iceberg-rest\\.default-catalog-name"
+
+ - it: renders log rotation and retention defaults
+ release:
+ name: gravitino
+ asserts:
+ - matchRegex:
+ path: data["log4j2.properties"]
+ pattern: "property\\.logMaxAge = 30d"
+ - matchRegex:
+ path: data["log4j2.properties"]
+ pattern: "property\\.serverLogMaxTotalSize = 2GB"
+ - matchRegex:
+ path: data["log4j2.properties"]
+ pattern: "appender\\.rolling\\.filePattern =
\\$\\{basePath\\}/\\$\\{serverName\\}_%d\\{yyyyMMdd\\}\\.%i\\.log\\.gz"
+ - matchRegex:
+ path: data["log4j2.properties"]
+ pattern: "appender\\.rolling\\.policies\\.size\\.size = 100MB"
+ - matchRegex:
+ path: data["log4j2.properties"]
+ pattern: "appender\\.rolling\\.strategy\\.fileIndex = nomax"
+ - matchRegex:
+ path: data["log4j2.properties"]
+ pattern: "appender\\.rolling\\.strategy\\.delete\\.maxDepth = 1"
+ - matchRegex:
+ path: data["log4j2.properties"]
+ pattern: "appender\\.rolling\\.strategy\\.delete\\.ifFileName\\.glob
= \\$\\{serverName\\}_\\*\\.log\\.gz"
+ - matchRegex:
+ path: data["log4j2.properties"]
+ pattern:
"appender\\.rolling\\.strategy\\.delete\\.ifFileName\\.ifAny\\.ifAccumulatedFileSize\\.exceeds
= \\$\\{serverLogMaxTotalSize\\}"
+ - matchRegex:
+ path: data["log4j2.properties"]
+ pattern:
"appender\\.rolling\\.strategy\\.delete\\.ifFileName\\.ifAny\\.ifLastModified\\.age
= \\$\\{logMaxAge\\}"
+
+ - it: renders customized log retention
+ release:
+ name: gravitino
+ set:
+ log4j2Properties:
+ logMaxAge: 7d
+ serverLogMaxTotalSize: 500MB
+ rollingAppenderPoliciesSizeSize: 20MB
+ rollingAppenderStrategyDeleteIfFileNameGlob: "app_*.log.gz"
+ asserts:
+ - matchRegex:
+ path: data["log4j2.properties"]
+ pattern: "property\\.logMaxAge = 7d"
+ - matchRegex:
+ path: data["log4j2.properties"]
+ pattern: "property\\.serverLogMaxTotalSize = 500MB"
+ - matchRegex:
+ path: data["log4j2.properties"]
+ pattern: "appender\\.rolling\\.policies\\.size\\.size = 20MB"
+ - matchRegex:
+ path: data["log4j2.properties"]
+ pattern: "appender\\.rolling\\.strategy\\.delete\\.ifFileName\\.glob
= app_\\*\\.log\\.gz"
diff --git a/dev/charts/gravitino/values.yaml b/dev/charts/gravitino/values.yaml
index 40c72d86ad..b18b043a87 100644
--- a/dev/charts/gravitino/values.yaml
+++ b/dev/charts/gravitino/values.yaml
@@ -448,29 +448,37 @@ log4j2Properties: {}
# basePath: "${sys:gravitino.log.path}"
# serverName: "${sys:gravitino.server.name}"
+ ## Log retention. Rolled archives are kept for logMaxAge. Their total size
is also capped: when the
+ ## cap is exceeded, the oldest archives are deleted first.
+ # logMaxAge: 30d
+ # serverLogMaxTotalSize: 2GB
+
## RollingFileAppender name, pattern, path and rollover policy
# rollingAppenderType: RollingFile
# rollingAppenderName: fileLogger
# rollingAppenderFileName: "${basePath}/${serverName}.log"
- # rollingAppenderFilePattern: "${basePath}/${serverName}_%d{yyyyMMdd}.log.gz"
+ # rollingAppenderFilePattern:
"${basePath}/${serverName}_%d{yyyyMMdd}.%i.log.gz"
+ ## When you change the archive name, set
rollingAppenderStrategyDeleteIfFileNameGlob to match it.
# rollingAppenderLayoutType: PatternLayout
# rollingAppenderLayoutPattern: "%d{yyyy-MM-dd HH:mm:ss.SSS} %level [%t]
[%l] - %msg%n"
# rollingAppenderPoliciesType: Policies
## RollingFileAppender rotation policy
# rollingAppenderPoliciesSizeType: SizeBasedTriggeringPolicy
- # rollingAppenderPoliciesSizeSize: 10MB
+ # rollingAppenderPoliciesSizeSize: 100MB
# rollingAppenderPoliciesTimeType: TimeBasedTriggeringPolicy
# rollingAppenderPoliciesTimeInterval: 1
# rollingAppenderPoliciesTimeModulate: true
# rollingAppenderStrategyType: DefaultRolloverStrategy
+ # rollingAppenderStrategyFileIndex: nomax
# rollingAppenderStrategyDeleteType: Delete
# rollingAppenderStrategyDeleteBasePath: "${basePath}"
- # rollingAppenderStrategyDeleteMaxDepth: 10
+ # rollingAppenderStrategyDeleteMaxDepth: 1
+ # rollingAppenderStrategyDeleteIfFileNameGlob: "${serverName}_*.log.gz"
# rollingAppenderStrategyDeleteIfLastModifiedType: IfLastModified
- ## Delete all files older than 30 days
- # rollingAppenderStrategyDeleteIfLastModifiedAge: 30d
+ ## Delete archives older than this age. Defaults to logMaxAge.
+ # rollingAppenderStrategyDeleteIfLastModifiedAge: "${logMaxAge}"
## JCasbin logger
# jcasbinLoggerLevel: warn
diff --git a/docs/gravitino-server-config.md b/docs/gravitino-server-config.md
index d8421e6f04..3297ba7150 100644
--- a/docs/gravitino-server-config.md
+++ b/docs/gravitino-server-config.md
@@ -434,17 +434,27 @@ frequently — this can measurably increase audit log
volume; size log rotation
`FileAuditWriter` is the default writer, and it manages no files itself.
Rotation, compression, and
retention are delegated to Log4j2 through a logger named `gravitino.audit`,
configured by the
`audit_file` appender group in `conf/log4j2.properties`. Out of the box it
writes
-`gravitino_audit.log` under the log directory, rotates daily and at 256 MB,
gzips what it rotates,
-and deletes anything older than 30 days. Change the path or the retention
there:
+`gravitino_audit.log` under the log directory and rotates it daily and at 256
MB into numbered gzip
+archives. It deletes archives older than 30 days and, oldest first, archives
beyond 10 GB in total.
+Change the retention or the path there:
```properties
# conf/log4j2.properties
+property.auditLogMaxTotalSize = 30GB
+appender.audit_file.strategy.delete.ifFileName.ifAny.ifLastModified.age = 90d
+
appender.audit_file.fileName = /var/log/gravitino/my_audit.log
appender.audit_file.filePattern =
/var/log/gravitino/my_audit_%d{yyyyMMdd}.%i.log.gz
-
-appender.audit_file.strategy.delete.ifAll.ifLastModified.age = 90d
+# Deletion must look in the new directory and match the new archive names.
+appender.audit_file.strategy.delete.basePath = /var/log/gravitino
+appender.audit_file.strategy.delete.ifFileName.glob = my_audit_*.log.gz
```
+Earlier releases set the audit retention with
+`appender.audit_file.strategy.delete.ifAll.ifLastModified.age`. That key no
longer exists. Log4j2
+rejects a configuration file that still sets it, and the server then writes no
log files. See
+[Log rotation and retention](./how-to-install.md#log-rotation-and-retention)
for all logs.
+
Earlier releases configured the writer directly through
`gravitino.audit.writer.file.*`. Those
properties now do nothing, and `FileAuditWriter` logs a warning at startup if
it finds any of them.
diff --git a/docs/how-to-install.md b/docs/how-to-install.md
index 08a9bb4e5a..3ebeb83b91 100644
--- a/docs/how-to-install.md
+++ b/docs/how-to-install.md
@@ -75,6 +75,31 @@ The Gravitino server configuration file is
`conf/gravitino.conf`. Configure the
The Gravitino server log configuration file is `conf/log4j2.properties`.
Gravitino uses Log4j2 as the logging system. Refer to the [Log4j2 configuration
guide](https://logging.apache.org/log4j/2.x/) to do the log configuration.
+##### Log rotation and retention
+
+A log rotates when its current file reaches the roll size or when the day
ends. Every rotation compresses the file into its own archive, `<log
name>_<yyyyMMdd>.<index>.log.gz`, so no archive replaces another.
+
+When a log rotates, an archive of that log is deleted if it is older than
`logMaxAge`, or if the archives of the log exceed its total size cap. The
oldest archives are deleted first. Deletion only looks at the archives of the
log itself, directly in the log directory. The cap counts archives only, not
the file being written.
+
+| Log file | Configuration file
| Roll size | Kept for | Total size cap |
+|-------------------------------------|-------------------------------------------------|-----------|----------|---------------------------------|
+| `gravitino-server.log` | `conf/log4j2.properties`
| 100MB | 30 days | 2GB (`serverLogMaxTotalSize`) |
+| `gravitino_audit.log` | `conf/log4j2.properties`
| 256MB | 30 days | 10GB (`auditLogMaxTotalSize`) |
+| `gravitino_lineage.log` | `conf/log4j2.properties`
| 100MB | 30 days | 1GB (`lineageLogMaxTotalSize`) |
+| `gravitino-iceberg-rest-server.log` |
`conf/gravitino-iceberg-rest-log4j2.properties` | 100MB | 30 days | 2GB
(`serverLogMaxTotalSize`) |
+| `gravitino-lance-rest-server.log` |
`conf/gravitino-lance-rest-log4j2.properties` | 100MB | 30 days | 1GB
(`serverLogMaxTotalSize`) |
+
+The retention properties are at the top of each configuration file. For
example, to keep 90 days of logs with up to 5GB of server log archives:
+
+```properties
+property.logMaxAge = 90d
+property.serverLogMaxTotalSize = 5GB
+```
+
+`logMaxAge` applies to every log in the same file. The deletion rules are
Log4j2 [Delete
action](https://logging.apache.org/log4j/2.x/manual/appenders/rolling-file.html#DeleteAction)
conditions, and each `if...` key is named after a condition type: `ifFileName`
(`IfFileName`) selects the archives of the log by name, and `ifAny` (`IfAny`)
deletes an archive when either `ifLastModified` (`IfLastModified`, older than
an age) or `ifAccumulatedFileSize` (`IfAccumulatedFileSize`, beyond the si [...]
+
+`bin/gravitino.sh start` also writes the standard output and standard error of
the process to `logs/gravitino-server.out`. Each `start` rotates this file
first and keeps the previous files as `gravitino-server.out.1` (newest) to
`gravitino-server.out.5`. Set `GRAVITINO_OUT_FILE_KEEP` in
`conf/gravitino-env.sh` to keep a different number, or `0` to keep none.
+
#### Configure the Server Environment
The Gravitino server environment configuration file is
`conf/gravitino-env.sh`. Gravitino exposes several environment variables.
Modify them in this file.
diff --git
a/server/src/test/java/org/apache/gravitino/server/TestLog4j2RollingConfiguration.java
b/server/src/test/java/org/apache/gravitino/server/TestLog4j2RollingConfiguration.java
new file mode 100644
index 0000000000..ce2d6496ce
--- /dev/null
+++
b/server/src/test/java/org/apache/gravitino/server/TestLog4j2RollingConfiguration.java
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.server;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.BufferedReader;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.nio.file.attribute.FileTime;
+import java.time.Instant;
+import java.time.temporal.ChronoUnit;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+import java.util.zip.GZIPInputStream;
+import org.apache.logging.log4j.core.LoggerContext;
+import org.apache.logging.log4j.core.config.Configuration;
+import org.apache.logging.log4j.core.config.ConfigurationSource;
+import
org.apache.logging.log4j.core.config.properties.PropertiesConfigurationFactory;
+import org.junit.jupiter.api.io.TempDir;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+/**
+ * Verifies the rolling file appenders of the packaged log4j2 templates: every
roll is kept as its
+ * own archive, and archives are removed only by age or by the per-log total
size cap, and only the
+ * archives of the log itself.
+ */
+class TestLog4j2RollingConfiguration {
+
+ private static final String ROLL_SIZE = "1KB";
+ private static final int MESSAGE_COUNT = 400;
+ private static final Pattern MESSAGE_ID = Pattern.compile("message-(\\d+)-");
+
+ @TempDir Path logDir;
+
+ static Stream<Arguments> rollingLogs() {
+ return Stream.of(
+ Arguments.of(
+ "conf/log4j2.properties.template",
+ "gravitino-server",
+ "org.apache.gravitino.TestRootLogger",
+ "gravitino-server",
+ "serverLogMaxTotalSize"),
+ Arguments.of(
+ "conf/log4j2.properties.template",
+ "gravitino-server",
+ "gravitino.audit",
+ "gravitino_audit",
+ "auditLogMaxTotalSize"),
+ Arguments.of(
+ "conf/log4j2.properties.template",
+ "gravitino-server",
+ "org.apache.gravitino.lineage.sink.LineageLogSink$LineageLogger",
+ "gravitino_lineage",
+ "lineageLogMaxTotalSize"),
+ Arguments.of(
+ "conf/gravitino-iceberg-rest-log4j2.properties.template",
+ "gravitino-iceberg-rest-server",
+ "org.apache.gravitino.TestRootLogger",
+ "gravitino-iceberg-rest-server",
+ "serverLogMaxTotalSize"),
+ Arguments.of(
+ "conf/gravitino-lance-rest-log4j2.properties.template",
+ "gravitino-lance-rest-server",
+ "org.apache.gravitino.TestRootLogger",
+ "gravitino-lance-rest-server",
+ "serverLogMaxTotalSize"));
+ }
+
+ @ParameterizedTest(name = "{3}")
+ @MethodSource("rollingLogs")
+ void testEveryRollIsKeptAndOnlyOwnExpiredArchivesAreDeleted(
+ String template, String serverName, String loggerName, String logName,
String capProperty)
+ throws IOException {
+ FileTime expired = FileTime.from(Instant.now().minus(60, ChronoUnit.DAYS));
+ Path ownExpiredArchive = createArchive(logDir.resolve(logName +
"_20200101.1.log.gz"), expired);
+ Path otherExpiredArchive =
+ createArchive(logDir.resolve("other-log_20200101.1.log.gz"), expired);
+ Path nestedExpiredArchive =
+ createArchive(logDir.resolve("nested").resolve(logName +
"_20200101.1.log.gz"), expired);
+
+ writeMessages(template, serverName, loggerName, capProperty, "1GB");
+
+ Map<Integer, Path> archives = todayArchives(logName);
+ assertTrue(archives.size() > 1, "Expected multiple archives, but found " +
archives.keySet());
+ assertEquals(
+ IntStream.rangeClosed(1,
archives.size()).boxed().collect(Collectors.toList()),
+ new ArrayList<>(archives.keySet()),
+ "Every roll must be kept as its own archive");
+
+ List<Integer> ids = new ArrayList<>();
+ for (Path archive : archives.values()) {
+ ids.addAll(readMessageIds(archive));
+ }
+ ids.addAll(readMessageIds(logDir.resolve(logName + ".log")));
+ assertEquals(
+ IntStream.range(0, MESSAGE_COUNT).boxed().collect(Collectors.toList()),
+ ids.stream().sorted().collect(Collectors.toList()),
+ "Every message must be kept exactly once");
+
+ assertFalse(Files.exists(ownExpiredArchive), "Expired archive of the log
must be deleted");
+ assertTrue(Files.exists(otherExpiredArchive), "Archives of other logs must
be kept");
+ assertTrue(Files.exists(nestedExpiredArchive), "Files in subdirectories
must be kept");
+ }
+
+ @ParameterizedTest(name = "{3}")
+ @MethodSource("rollingLogs")
+ void testOldestArchivesAreDeletedBeyondTotalSizeCap(
+ String template, String serverName, String loggerName, String logName,
String capProperty)
+ throws IOException {
+ long capBytes = 4096;
+ writeMessages(template, serverName, loggerName, capProperty, capBytes /
1024 + "KB");
+
+ TreeMap<Integer, Path> archives = todayArchives(logName);
+ assertFalse(archives.isEmpty(), "Expected rolled archives of " + logName);
+ long totalSize = 0;
+ for (Path archive : archives.values()) {
+ totalSize += Files.size(archive);
+ }
+ assertTrue(totalSize <= capBytes, "Archives use " + totalSize + " bytes
beyond the cap");
+ assertFalse(archives.containsKey(1), "The oldest archive must be deleted
first");
+ int newestIndex = archives.lastKey();
+ assertTrue(
+ newestIndex > archives.size(), "Expected deleted archives, but found "
+ archives.keySet());
+ }
+
+ private void writeMessages(
+ String template, String serverName, String loggerName, String
capProperty, String cap)
+ throws IOException {
+ String content =
+ new String(Files.readAllBytes(rootDir().resolve(template)),
StandardCharsets.UTF_8);
+ content = replaceValue(content, "property\\.basePath", logDir.toString());
+ content = replaceValue(content, "property\\.serverName", serverName);
+ content = replaceValue(content, "property\\." + capProperty, cap);
+ content = replaceValue(content,
"appender\\.\\w+\\.policies\\.size\\.size", ROLL_SIZE);
+
+ LoggerContext context = new LoggerContext("log4j2-rolling-test-" +
UUID.randomUUID());
+ try (InputStream input = new
ByteArrayInputStream(content.getBytes(StandardCharsets.UTF_8))) {
+ Configuration configuration =
+ new PropertiesConfigurationFactory()
+ .getConfiguration(context, new ConfigurationSource(input));
+ context.start(configuration);
+ }
+
+ try {
+ for (int i = 0; i < MESSAGE_COUNT; i++) {
+ // Random content keeps the gzip archives close to the roll size.
+ context
+ .getLogger(loggerName)
+ .info("message-{}-{}{}", i, UUID.randomUUID(), UUID.randomUUID());
+ }
+ } finally {
+ assertTrue(context.stop(30, TimeUnit.SECONDS), "Log4j2 context did not
stop in time");
+ }
+ }
+
+ private TreeMap<Integer, Path> todayArchives(String logName) throws
IOException {
+ // Skip the expired archive that a test creates before logging.
+ Pattern archiveName =
+ Pattern.compile(Pattern.quote(logName) +
"_(?!20200101)\\d{8}\\.(\\d+)\\.log\\.gz");
+ TreeMap<Integer, Path> archives = new TreeMap<>();
+ try (Stream<Path> files = Files.list(logDir)) {
+ for (Path file : files.collect(Collectors.toList())) {
+ Matcher matcher = archiveName.matcher(file.getFileName().toString());
+ if (matcher.matches()) {
+ archives.put(Integer.parseInt(matcher.group(1)), file);
+ }
+ }
+ }
+ return archives;
+ }
+
+ private static String replaceValue(String content, String keyRegex, String
value) {
+ Matcher matcher = Pattern.compile("(?m)^(" + keyRegex +
"\\s*=\\s*).*$").matcher(content);
+ assertTrue(matcher.find(), "Missing property " + keyRegex);
+ return matcher.replaceAll("$1" + Matcher.quoteReplacement(value));
+ }
+
+ private static Path createArchive(Path archive, FileTime lastModified)
throws IOException {
+ Files.createDirectories(archive.getParent());
+ Files.write(archive, new byte[] {0});
+ Files.setLastModifiedTime(archive, lastModified);
+ return archive;
+ }
+
+ private static List<Integer> readMessageIds(Path file) throws IOException {
+ List<Integer> ids = new ArrayList<>();
+ if (!Files.exists(file)) {
+ return ids;
+ }
+
+ InputStream input = Files.newInputStream(file);
+ if (file.toString().endsWith(".gz")) {
+ input = new GZIPInputStream(input);
+ }
+ try (BufferedReader reader =
+ new BufferedReader(new InputStreamReader(input,
StandardCharsets.UTF_8))) {
+ String line;
+ while ((line = reader.readLine()) != null) {
+ Matcher matcher = MESSAGE_ID.matcher(line);
+ if (matcher.find()) {
+ ids.add(Integer.parseInt(matcher.group(1)));
+ }
+ }
+ }
+ return ids;
+ }
+
+ static Path rootDir() {
+ String home = System.getenv("GRAVITINO_HOME");
+ if (home != null && Files.exists(Paths.get(home, "conf",
"log4j2.properties.template"))) {
+ return Paths.get(home);
+ }
+ return Paths.get("").toAbsolutePath().getParent();
+ }
+}
diff --git
a/server/src/test/java/org/apache/gravitino/server/TestOutFileRotation.java
b/server/src/test/java/org/apache/gravitino/server/TestOutFileRotation.java
new file mode 100644
index 0000000000..08215311cf
--- /dev/null
+++ b/server/src/test/java/org/apache/gravitino/server/TestOutFileRotation.java
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.server;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.Nullable;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+/** Verifies {@code rotate_out_file} in {@code bin/common.sh.template}. */
+class TestOutFileRotation {
+
+ @TempDir Path dir;
+
+ @Test
+ void testKeepsFivePreviousFilesByDefault() throws Exception {
+ Path outFile = dir.resolve("gravitino-server.out");
+ for (int run = 0; run < 7; run++) {
+ write(outFile, "run-" + run);
+ rotate(outFile, null);
+ }
+
+ assertFalse(Files.exists(outFile));
+ for (int index = 1; index <= 5; index++) {
+ assertEquals("run-" + (7 - index), read(outFile, index));
+ }
+ assertFalse(Files.exists(rotated(outFile, 6)));
+ }
+
+ @Test
+ void testRemovesFilesBeyondCustomLimit() throws Exception {
+ Path outFile = dir.resolve("gravitino-server.out");
+ write(outFile, "current");
+ for (int index = 1; index <= 6; index++) {
+ write(rotated(outFile, index), "previous-" + index);
+ }
+
+ rotate(outFile, "2");
+
+ assertFalse(Files.exists(outFile));
+ assertEquals("current", read(outFile, 1));
+ assertEquals("previous-1", read(outFile, 2));
+ for (int index = 3; index <= 7; index++) {
+ assertFalse(Files.exists(rotated(outFile, index)));
+ }
+ }
+
+ @Test
+ void testKeepsNoPreviousFileWhenLimitIsZero() throws Exception {
+ Path outFile = dir.resolve("gravitino-server.out");
+ write(outFile, "current");
+ write(rotated(outFile, 1), "previous-1");
+
+ rotate(outFile, "0");
+
+ assertFalse(Files.exists(outFile));
+ assertFalse(Files.exists(rotated(outFile, 1)));
+ }
+
+ @Test
+ void testFallsBackToDefaultLimitWhenLimitIsInvalid() throws Exception {
+ Path outFile = dir.resolve("gravitino-server.out");
+ for (int index = 1; index <= 5; index++) {
+ write(rotated(outFile, index), "previous-" + index);
+ }
+ write(outFile, "current");
+
+ rotate(outFile, "abc");
+
+ assertEquals("current", read(outFile, 1));
+ assertEquals("previous-4", read(outFile, 5));
+ assertFalse(Files.exists(rotated(outFile, 6)));
+ }
+
+ @Test
+ void testDoesNothingWithoutOutFile() throws Exception {
+ Path outFile = dir.resolve("gravitino-server.out");
+
+ rotate(outFile, null);
+
+ assertFalse(Files.exists(outFile));
+ assertFalse(Files.exists(rotated(outFile, 1)));
+ }
+
+ private void rotate(Path outFile, @Nullable String keep) throws Exception {
+ ProcessBuilder builder =
+ new ProcessBuilder(
+ "bash", "-c", ". \"${COMMON_SH}\" > /dev/null && rotate_out_file
\"${OUT_FILE}\"");
+ builder.redirectErrorStream(true);
+ Map<String, String> env = builder.environment();
+ env.put(
+ "COMMON_SH",
+
TestLog4j2RollingConfiguration.rootDir().resolve("bin/common.sh.template").toString());
+ env.put("OUT_FILE", outFile.toString());
+ env.put("GRAVITINO_HOME", dir.toString());
+ env.put("GRAVITINO_CONF_DIR", dir.resolve("conf").toString());
+ env.put("GRAVITINO_LOG_DIR", dir.toString());
+ env.put("GRAVITINO_VERSION", "test");
+ env.remove("GRAVITINO_OUT_FILE_KEEP");
+ if (keep != null) {
+ env.put("GRAVITINO_OUT_FILE_KEEP", keep);
+ }
+
+ Process process = builder.start();
+ String output;
+ try (InputStream input = process.getInputStream()) {
+ ByteArrayOutputStream buffer = new ByteArrayOutputStream();
+ input.transferTo(buffer);
+ output = buffer.toString(StandardCharsets.UTF_8);
+ }
+ assertTrue(process.waitFor(30, TimeUnit.SECONDS), "rotate_out_file timed
out");
+ assertEquals(0, process.exitValue(), "rotate_out_file failed: " + output);
+ }
+
+ private static Path rotated(Path outFile, int index) {
+ return outFile.resolveSibling(outFile.getFileName() + "." + index);
+ }
+
+ private static void write(Path file, String content) throws IOException {
+ Files.write(file, content.getBytes(StandardCharsets.UTF_8));
+ }
+
+ private static String read(Path outFile, int index) throws IOException {
+ return new String(Files.readAllBytes(rotated(outFile, index)),
StandardCharsets.UTF_8);
+ }
+}
