Haram Lee created SPARK-61742:
---------------------------------

             Summary: Open event logs with the sequential read policy
                 Key: SPARK-61742
                 URL: https://issues.apache.org/jira/browse/SPARK-61742
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 5.0.0
            Reporter: Haram Lee


{{EventLogFileReader.openEventLog}} opens event logs with 
{{FileSystem.open()}}. Event logs are always read from start to end, but this 
call does not tell the filesystem about that.

Since HADOOP-19559 (Hadoop 3.4.3 and 3.5.0), the default S3A input stream is 
the analytics stream. It keeps per-object state for the lifetime of the 
filesystem, and only removes an object's state when the stream is closed if the 
stream was opened with a sequential read policy (see HADOOP-20058). A History 
Server that reads event logs from S3 keeps one cached FileSystem for its whole 
life, so its heap grows with every event log it reads, until it runs out of 
memory.

A local measurement against an S3-compatible server, reading 4,000 small event 
logs with hadoop-aws 3.5.0 and the default stream type:
* {{FileSystem.open()}}: live heap grew by about 34 KB per log, and 144,100 
{{MeanStatistic}} objects were retained.
* {{FileSystem.openFile()}} with {{fs.option.openfile.read.policy=sequential}}: 
live heap did not grow.

Proposed fix: open event logs with {{FileSystem.openFile()}} and set 
{{fs.option.openfile.read.policy}} to {{sequential}}, the same way 
{{HadoopLineRecordReader}} already uses {{openFile()}} (SPARK-52482).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to