wangyum opened a new pull request, #57796:
URL: https://github.com/apache/spark/pull/57796
### What changes were proposed in this pull request?
`ApplicationInfo` and `DriverInfo` are tagged with the active `SparkConf` at
the two
places `Master` constructs them (`createApplication`/`createDriver`), and a
`writeReplace()` hook is added to both classes so that, whenever an instance
is
actually Java-serialized (i.e. by `PersistenceEngine` when persisting Master
HA
recovery state to ZooKeeper, the filesystem, or RocksDB), a redacted copy is
substituted in place of the original. Redaction reuses the existing
`Utils.redact`
/ `Utils.redactCommandLineArgs` helpers already used elsewhere (e.g.
`JsonProtocol.writeApplicationDescription`) to scrub the app/driver
`Command`'s
environment variables and `-D` Java options.
The in-memory objects used by the running `Master` are untouched; only the
bytes
that get written to the recovery store are redacted.
### Why are the changes needed?
`PersistenceEngine` implementations persist raw
`ApplicationInfo`/`DriverInfo`
objects via plain Java serialization for Standalone Master HA recovery. If an
application or driver's launch command contains secrets in its environment
variables or Java options (e.g. passwords, tokens), those secrets are
currently
written in plaintext to ZooKeeper znodes or the on-disk recovery directory,
where they can persist indefinitely and be read by anyone with access to that
store.
### Does this PR introduce _any_ user-facing change?
No. The Master UI/REST JSON output and RPC responses are unchanged; the fix
only
affects the bytes written to the HA recovery store
(ZooKeeper/filesystem/RocksDB),
which are not directly user-visible.
### How was this patch tested?
Added tests:
- `JsonProtocolSuite`: `SPARK-58592: redactedCopy redacts secrets in
ApplicationInfo
and DriverInfo` verifies the redaction helper directly; `SPARK-58592:
writeReplace
redacts secrets when Java-serialized for persistence` round-trips tagged
instances
through a real `JavaSerializer` and asserts secrets are redacted
post-deserialize
while non-secret fields survive.
- `PersistenceEngineSuite`: `SPARK-58592: FileSystemPersistenceEngine
redacts secrets
in ApplicationInfo and DriverInfo` persists a secret-bearing
`ApplicationInfo`/
`DriverInfo` through a real `FileSystemPersistenceEngine`, reads the raw
bytes
written to disk to assert the plaintext secret is absent, and reads the
objects
back via `PersistenceEngine.read` to confirm redacted values and intact
non-secret
fields.
Ran:
```
build/sbt "core/testOnly org.apache.spark.deploy.JsonProtocolSuite
org.apache.spark.deploy.master.PersistenceEngineSuite"
```
All 22 tests pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (claude-sonnet-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
