HeartSaVioR edited a comment on issue #24128: [SPARK-27188][SS] FileStreamSink:
provide a new option to have retention on output files
URL: https://github.com/apache/spark/pull/24128#issuecomment-558976957
> IMHO, the core problem is the compact metadata log grows bigger and
bigger, and it
HeartSaVioR edited a comment on issue #24128: [SPARK-27188][SS] FileStreamSink:
provide a new option to have retention on output files
URL: https://github.com/apache/spark/pull/24128#issuecomment-558976957
> IMHO, the core problem is the compact metadata log grows bigger and
bigger, and it
HeartSaVioR edited a comment on issue #24128: [SPARK-27188][SS] FileStreamSink:
provide a new option to have retention on output files
URL: https://github.com/apache/spark/pull/24128#issuecomment-557398360
UPDATE: SPARK-29995 is just filed from other end user which denotes same
issues SPAR
HeartSaVioR edited a comment on issue #24128: [SPARK-27188][SS] FileStreamSink:
provide a new option to have retention on output files
URL: https://github.com/apache/spark/pull/24128#issuecomment-474149223
This is a commit based on master for the last alternative (exclude old
output files
HeartSaVioR edited a comment on issue #24128: [SPARK-27188][SS] FileStreamSink:
provide a new option to have retention on output files
URL: https://github.com/apache/spark/pull/24128#issuecomment-474153829
Rebased to the approach: applying retention. Also updated JIRA and PR as
well.
HeartSaVioR edited a comment on issue #24128: [SPARK-27188][SS] FileStreamSink:
provide a new option to have retention on output files
URL: https://github.com/apache/spark/pull/24128#issuecomment-474150802
> We shouldn't implement ad-hoc changes with unclear behavior and semantics
just to
