[ 
https://issues.apache.org/jira/browse/HDDS-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Yarovoy updated HDDS-17857:
----------------------------------
    Description: 
h3. Problem

Several tests turn on audit logging by setting 
\{{log4j.configurationFile=auditlog.properties}}. Every 
\{{auditlog.properties}} under \{{src/test/resources}} names its file with a 
relative path:

{noformat}
appender.audit.fileName=audit.log
appender.systemaudit.fileName=system_audit.log (integration-test only)
{noformat}

Log4j resolves these paths against the working directory of the surefire JVM, 
which is the module directory, so the files land next to \{{pom.xml}} and 
\{{src/}}, not under \{{target/}}. They show up as untracked files in \{{git 
status}} (\{{.gitignore}} doesn't cover them), and \{{mvn clean}} doesn't 
remove them, so they pile up between runs. A careless \{{git add -A}} can 
commit them.

Example: after running \{{TestSecureOzoneRpcClient}}, \{{git status}} shows

{noformat}
?? hadoop-ozone/integration-test/audit.log
?? hadoop-ozone/integration-test/system_audit.log
{noformat}

{\{OzoneRpcClientTests}} (the base of \{{TestOzoneRpcClient}} and 
\{{TestSecureOzoneRpcClient}}) calls \{{AuditLogTestUtils.enableAuditLog()}} 
but never deletes the files. Other tests clean up only partly: \{{TestOmAcls}} 
and \{{TestOMRatisSnapshotTransfer}} delete them, and \{{TestOzoneAuditLogger}} 
deletes \{{audit.log}} in \{{@AfterAll}}. That cleanup doesn't run if the fork 
is killed or times out.

Affected configs (only the integration-test one also writes 
\{{system_audit.log}}):

{noformat}
hadoop-ozone/integration-test/src/test/resources/auditlog.properties
hadoop-hdds/framework/src/test/resources/auditlog.properties
hadoop-hdds/rocks-native/src/test/resources/auditlog.properties
hadoop-ozone/s3gateway/src/test/resources/auditlog.properties
{noformat}

The readers use the same relative names: \{{AuditLogTestUtils}} 
(\{{AUDITLOG_FILENAME}}, \{{SYSTEM_AUDITLOG_FILENAME}}), 
\{{TestOzoneAuditLogger}}, \{{TestS3GatewayAuditLog}} and 
\{{TestOzoneRpcClientForAclAuditLog}}.

h3. Proposed fix

Write the files under the module's \{{target/}} directory. Surefire already 
sets the system property \{{hadoop.log.dir}} to the module's \{{target/log}} 
directory in every test JVM (the \{{systemPropertyVariables}} of the surefire 
plugin in the root \{{pom.xml}}), so:

# In each test \{{auditlog.properties}}, use a log4j2 system property lookup 
with a fallback for IDE runs, e.g.
{noformat}
appender.audit.fileName=${sys:hadoop.log.dir:-target/log}/audit.log
appender.systemaudit.fileName=${sys:hadoop.log.dir:-target/log}/system_audit.log
{noformat}
# Resolve the same location in \{{AuditLogTestUtils}} (e.g. 
\{{Paths.get(System.getProperty("hadoop.log.dir", "target/log"), 
"audit.log")}}) and use it in the tests that open the file directly 
(\{{TestOzoneAuditLogger}}, \{{TestS3GatewayAuditLog}}, 
\{{TestOzoneRpcClientForAclAuditLog}}), so that writers and readers agree.
# Optionally, add \{{audit.log}} and \{{system_audit.log}} to \{{.gitignore}}, 
so that files left by older runs or branches don't show up.

h3. Compatibility / side effects

Test-only. The production audit configs in 
\{{hadoop-ozone/dist/src/shell/conf/*-audit-log4j2.properties}} don't change. 
\{{mvn clean}} will now remove the files.

h3. Out of scope

Tests that run in parallel forks of the same module still share one file, 
because they share one \{{target/log}}. Giving each fork its own directory 
(e.g. with the \{{surefire.forkNumber}} property) would be a separate change.

h3. Tests

* Run the audit tests listed above and check that they still pass.
* After the run, \{{git status}} shows no \{{audit.log}} or 
\{{system_audit.log}}, and the files are under \{{<module>/target/log/}}.

  was:
h3. Problem

Several tests turn on audit logging by setting 
\{{log4j.configurationFile=auditlog.properties}}. Every 
\{{auditlog.properties}} under \{{src/test/resources}} names its file with a 
relative path:

{noformat}
appender.audit.fileName=audit.log
appender.systemaudit.fileName=system_audit.log (integration-test only)
{noformat}

Log4j resolves these paths against the working directory of the surefire JVM, 
which is the module directory, so the files land next to \{{pom.xml}} and 
\{{src/}}, not under \{{target/}}. They show up as untracked files in \{{git 
status}} (\{{.gitignore}} doesn't cover them), and \{{mvn clean}} doesn't 
remove them, so they pile up between runs. A careless \{{git add -A}} can 
commit them.

Example: after running \{{TestSecureOzoneRpcClient}}, \{{git status}} shows

{noformat}
?? hadoop-ozone/integration-test/audit.log
?? hadoop-ozone/integration-test/system_audit.log
{noformat}

{\{OzoneRpcClientTests}} (the base of \{{TestOzoneRpcClient}} and 
\{{TestSecureOzoneRpcClient}}) calls \{{AuditLogTestUtils.enableAuditLog()}} 
but never deletes the files. Other tests clean up only partly: \{{TestOmAcls}} 
and \{{TestOMRatisSnapshotTransfer}} delete them, and \{{TestOzoneAuditLogger}} 
deletes \{{audit.log}} in \{{@AfterAll}}. That cleanup doesn't run if the fork 
is killed or times out.

Affected configs:
* \{{hadoop-ozone/integration-test/src/test/resources/auditlog.properties}} 
(\{{audit.log}}, \{{system_audit.log}})
* \{{hadoop-hdds/framework/src/test/resources/auditlog.properties}}
* \{{hadoop-hdds/rocks-native/src/test/resources/auditlog.properties}}
* \{{hadoop-ozone/s3gateway/src/test/resources/auditlog.properties}}

The readers use the same relative names: \{{AuditLogTestUtils}} 
(\{{AUDITLOG_FILENAME}}, \{{SYSTEM_AUDITLOG_FILENAME}}), 
\{{TestOzoneAuditLogger}}, \{{TestS3GatewayAuditLog}} and 
\{{TestOzoneRpcClientForAclAuditLog}}.

h3. Proposed fix

Write the files under the module's \{{target/}} directory. Surefire already 
passes \{{hadoop.log.dir=$\{project.build.directory\}/log}} to every test JVM 
(root \{{pom.xml}}, \{{systemPropertyVariables}}), so:

# In each test \{{auditlog.properties}}, use a log4j2 system property lookup 
with a fallback for IDE runs, e.g.
{noformat}
appender.audit.fileName=${sys:hadoop.log.dir:-target/log}/audit.log
appender.systemaudit.fileName=${sys:hadoop.log.dir:-target/log}/system_audit.log
{noformat}
# Resolve the same location in \{{AuditLogTestUtils}} (e.g. 
\{{Paths.get(System.getProperty("hadoop.log.dir", "target/log"), 
"audit.log")}}) and use it in the tests that open the file directly 
(\{{TestOzoneAuditLogger}}, \{{TestS3GatewayAuditLog}}, 
\{{TestOzoneRpcClientForAclAuditLog}}), so that writers and readers agree.
# Optionally, add \{{audit.log}} and \{{system_audit.log}} to \{{.gitignore}}, 
so that files left by older runs or branches don't show up.

h3. Compatibility / side effects

Test-only. The production audit configs in 
\{{hadoop-ozone/dist/src/shell/conf/*-audit-log4j2.properties}} don't change. 
\{{mvn clean}} will now remove the files.

h3. Out of scope

Tests that run in parallel forks of the same module still share one file, 
because they share one \{{target/log}}. Separating them (e.g. with 
\{{$\{surefire.forkNumber\}}}) would be a separate change.

h3. Tests

* Run the audit tests listed above and check that they still pass.
* After the run, \{{git status}} shows no \{{audit.log}} or 
\{{system_audit.log}}, and the files are under \{{<module>/target/log/}}.


> Tests write audit.log and system_audit.log into the module directory instead 
> of target
> --------------------------------------------------------------------------------------
>
>                 Key: HDDS-17857
>                 URL: https://issues.apache.org/jira/browse/HDDS-17857
>             Project: Apache Ozone
>          Issue Type: Bug
>          Components: test
>            Reporter: Andrey Yarovoy
>            Assignee: Andrey Yarovoy
>            Priority: Major
>
> h3. Problem
> Several tests turn on audit logging by setting 
> \{{log4j.configurationFile=auditlog.properties}}. Every 
> \{{auditlog.properties}} under \{{src/test/resources}} names its file with a 
> relative path:
> {noformat}
> appender.audit.fileName=audit.log
> appender.systemaudit.fileName=system_audit.log (integration-test only)
> {noformat}
> Log4j resolves these paths against the working directory of the surefire JVM, 
> which is the module directory, so the files land next to \{{pom.xml}} and 
> \{{src/}}, not under \{{target/}}. They show up as untracked files in \{{git 
> status}} (\{{.gitignore}} doesn't cover them), and \{{mvn clean}} doesn't 
> remove them, so they pile up between runs. A careless \{{git add -A}} can 
> commit them.
> Example: after running \{{TestSecureOzoneRpcClient}}, \{{git status}} shows
> {noformat}
> ?? hadoop-ozone/integration-test/audit.log
> ?? hadoop-ozone/integration-test/system_audit.log
> {noformat}
> {\{OzoneRpcClientTests}} (the base of \{{TestOzoneRpcClient}} and 
> \{{TestSecureOzoneRpcClient}}) calls \{{AuditLogTestUtils.enableAuditLog()}} 
> but never deletes the files. Other tests clean up only partly: 
> \{{TestOmAcls}} and \{{TestOMRatisSnapshotTransfer}} delete them, and 
> \{{TestOzoneAuditLogger}} deletes \{{audit.log}} in \{{@AfterAll}}. That 
> cleanup doesn't run if the fork is killed or times out.
> Affected configs (only the integration-test one also writes 
> \{{system_audit.log}}):
> {noformat}
> hadoop-ozone/integration-test/src/test/resources/auditlog.properties
> hadoop-hdds/framework/src/test/resources/auditlog.properties
> hadoop-hdds/rocks-native/src/test/resources/auditlog.properties
> hadoop-ozone/s3gateway/src/test/resources/auditlog.properties
> {noformat}
> The readers use the same relative names: \{{AuditLogTestUtils}} 
> (\{{AUDITLOG_FILENAME}}, \{{SYSTEM_AUDITLOG_FILENAME}}), 
> \{{TestOzoneAuditLogger}}, \{{TestS3GatewayAuditLog}} and 
> \{{TestOzoneRpcClientForAclAuditLog}}.
> h3. Proposed fix
> Write the files under the module's \{{target/}} directory. Surefire already 
> sets the system property \{{hadoop.log.dir}} to the module's \{{target/log}} 
> directory in every test JVM (the \{{systemPropertyVariables}} of the surefire 
> plugin in the root \{{pom.xml}}), so:
> # In each test \{{auditlog.properties}}, use a log4j2 system property lookup 
> with a fallback for IDE runs, e.g.
> {noformat}
> appender.audit.fileName=${sys:hadoop.log.dir:-target/log}/audit.log
> appender.systemaudit.fileName=${sys:hadoop.log.dir:-target/log}/system_audit.log
> {noformat}
> # Resolve the same location in \{{AuditLogTestUtils}} (e.g. 
> \{{Paths.get(System.getProperty("hadoop.log.dir", "target/log"), 
> "audit.log")}}) and use it in the tests that open the file directly 
> (\{{TestOzoneAuditLogger}}, \{{TestS3GatewayAuditLog}}, 
> \{{TestOzoneRpcClientForAclAuditLog}}), so that writers and readers agree.
> # Optionally, add \{{audit.log}} and \{{system_audit.log}} to 
> \{{.gitignore}}, so that files left by older runs or branches don't show up.
> h3. Compatibility / side effects
> Test-only. The production audit configs in 
> \{{hadoop-ozone/dist/src/shell/conf/*-audit-log4j2.properties}} don't change. 
> \{{mvn clean}} will now remove the files.
> h3. Out of scope
> Tests that run in parallel forks of the same module still share one file, 
> because they share one \{{target/log}}. Giving each fork its own directory 
> (e.g. with the \{{surefire.forkNumber}} property) would be a separate change.
> h3. Tests
> * Run the audit tests listed above and check that they still pass.
> * After the run, \{{git status}} shows no \{{audit.log}} or 
> \{{system_audit.log}}, and the files are under \{{<module>/target/log/}}.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to