Chandni Singh created SPARK-43179:
-------------------------------------
Summary: Allow applications to control whether their metadata gets
saved by the shuffle server in the db
Key: SPARK-43179
URL: https://issues.apache.org/jira/browse/SPARK-43179
Project: Spark
Issue Type: Improvement
Components: Shuffle
Affects Versions: 3.4.0
Reporter: Chandni Singh
Currently, the External Shuffle Service stores application metadata in LevelDB.
This is necessary to enable the shuffle server to resume serving shuffle data
for an application whose executors registered before the NodeManager restarts.
However, the metadata includes the application secret, which is stored in
LevelDB without encryption. This is a potential security risk, particularly for
applications with high security requirements. While filesystem access control
lists (ACLs) can help protect keys and certificates, they may not be sufficient
for some use cases. In response, we have decided not to store metadata for
these high-security applications in LevelDB. As a result, these applications
may experience more failures in the event of a node restart, but we believe
this trade-off is acceptable given the increased security risk.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
