Hi Jim, Thanks a lot for the detailed feedback and the helpful pointers.
[1][2] Yes, I have read through FLIP‑505 carefully, but it seems this work has not progressed further. I also emailed the authors regarding this discussion, but have not received a reply. Besides, from my understanding: - FLIP-505 focuses on remote store partial caching and per-job fetching (If the job has an excessive volume of archive files, users will still wait for all files are downloaded before visible) - This FLIP focuses on lazy parsing, overview/detail separation, and async loading These two approaches seem complementary rather than conflicting, and could potentially work well together. [3] Thanks for the suggestion — returning a String would indeed simplify the current implementation. However, after re-evaluating the design, I think keeping a generic return type is more appropriate for a pluggable storage abstraction, considering different storage backends may return different data types. [4] Good catch. In our production environment, the thread count is set relatively high to ensure the service works fast. However, regarding the randomness of user requests and applicability, there is indeed no need for the default configuration to be excessively large. The default thread count could be reduced to 4 or 8, and users could configure this setting to suit their specific requirements. What are your thoughts? Best regards, Zihao Jim Hughes via dev <[email protected]> 于2026年4月28日周二 02:03写道： > Hi Zihao, > > [1] Have you read through > > https://cwiki.apache.org/confluence/display/FLINK/FLIP+505%3A+Flink+History+Server+Scability+Improvements%2C+Remote+Data+Store+Fetch+and+Per+Job+Fetch > ? > How does it relate to this work? > > [2] There are some PRs for the FLIP-505 work which have been open for > awhile here: > https://github.com/apache/flink/pulls?q=is%3Apr+is%3Aopen+flip-505+37155 > > [3] Shouldn't the `get` method return a `String`? > > [4] I'm surprised by the high default thread counts for the lazy loader. I > suppose you have some experience tuning that number? > > The rest of it looks reasonable. > > Cheers, > > Jim > > On Mon, Mar 23, 2026 at 8:02 AM zihao chen <[email protected]> wrote: > > > Hi all, > > > > I’d like to start a discussion on *FLIP-XXX: Enhancing HistoryServer with > > Lazy Loading and Pluggable Storage*. > > > > This FLIP proposes improving the HistoryServer to address *slow > > startup*, *delayed > > job visibility*, and *excessive small files* when handling large numbers > of > > archived jobs. > > > > The proposal introduces: > > > > - *Lazy loading mode* for on-demand job archive fetching > > - Optional *RocksDB-based storage* to reduce small files > > > > Compatibility: Full backward compatibility (EAGER + FILE as default) > > > > The detailed designs are described in the FLIP document: > > > > > > > https://docs.google.com/document/d/1FQn-HQSIdhio3ImeOrbS9VfB6JqRTrRV4ZEZDTf3K2o/edit?usp=sharing > > > > Looking forward to your feedback. > > > > > > Best regards, > > > > Zihao Chen > > >
