[
https://issues.apache.org/jira/browse/SPARK-17513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15498425#comment-15498425
]
Apache Spark commented on SPARK-17513:
--------------------------------------
User 'petermaxlee' has created a pull request for this issue:
https://github.com/apache/spark/pull/15126
> StreamExecution should discard unneeded metadata
> ------------------------------------------------
>
> Key: SPARK-17513
> URL: https://issues.apache.org/jira/browse/SPARK-17513
> Project: Spark
> Issue Type: Sub-task
> Components: Streaming
> Reporter: Frederick Reiss
>
> The StreamExecution maintains a write-ahead log of batch metadata in order to
> allow repeating previously in-flight batches if the driver is restarted.
> StreamExecution does not garbage-collect or compact this log in any way.
> Since the log is implemented with HDFSMetadataLog, these files will consume
> memory on the HDFS NameNode. Specifically, each log file will consume about
> 300 bytes of NameNode memory (150 bytes for the inode and 150 bytes for the
> block of file contents; see
> [https://www.cloudera.com/documentation/enterprise/latest/topics/admin_nn_memory_config.html].
> An application with a 100 msec batch interval will increase the NameNode's
> heap usage by about 250MB per day.
> There is also the matter of recovery. StreamExecution reads its entire log
> when restarting. This read operation will be very expensive if the log
> contains millions of entries spread over millions of files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
