Hi Kartikey,
Thank you for the positive feedback and support for this FLIP!
Your long-term questions are very insightful and touch upon
crucial aspects of the proposal's vision and practical implementation.
1. Scope of Application HA
You have raised an excellent point about the scope of idempotency. In
my opinion, the vision is to focus on the Flink job submission part,
i.e. making the `env.execute()` idempotent. The reason is, Flink, as a
framework, cannot be aware of the arbitrary stateful logic and external
side effects within a user's `main()` method. Attempting to manage
them would be complex and potentially intrusive.
Therefore, the responsibility for managing the idempotency of such
external operations will remain with the user in their application logic.
This approach is consistent with Flink's existing behavior, ensuring
we don't introduce breaking changes or require users to adopt a new,
complex contract for their application code.
2. History Server performance for /jobs/:jobid lookups
Your concern about the performance of /jobs/:jobid lookups for
automated tools is also spot on. First, I'd like to clarify how the
History Server works with the new hierarchical archive structure.
Ignoring the automatic fetching feature from FLIP-505 for a moment,
the History Server operates by periodically scanning the archive
directory(specified by jobmanager.archive.fs.dir) and building its own
local index (typically in the directory specified by
historyserver.web.tmpdir). The key detail is that this generated index
"flattens" the job structure. A job's archive is stored directly at a path
like jobs/{job-id}.json within this index directory. While there might be
a minor overhead during the indexing process itself, the lookup
performance for a specific job ID should have a limited regression.
You are absolutely right to point out that the real challenge might
appear with the automatic fetching feature (FLIP-505). However, that
mechanism is designed with its own caching layer. A multi-level lookup
would typically only occur on the first request for a job (maybe after a
cache miss). Therefore, I believe the History Server's built-in cache
will be sufficient to provide the necessary acceleration for this critical
use case.
Once again, thank you for this fantastic and constructive feedback.
I'm happy to discuss this further if you have more thoughts.
Best,
Yi
At 2025-10-01 18:30:04, "Kartikey Pant" <[email protected]> wrote:
>Hi Yi,
>
>Thanks for the excellent work on this FLIP. Formalizing the Application
>concept is a huge step forward, and it will definitely make the History
>Server more intuitive for batch jobs.
>
>I was thinking through the follow-up tasks and had a couple of long-term
>questions.
>
>First, on Application HA: making env.execute() idempotent is the right
>goal. My question is about the scope - is the vision to also help users
>manage stateful logic in their main() with its external side effects, or is
>the focus for now just on the Flink job submission part?
>
>My other question is about the History Server. The new API is great for the
>UI, but how should automated tools handle fast lookups via /jobs/:jobid?
>I'm just wondering if the expectation is that operators will need to build
>their own index to maintain performance for that critical use case.
>
>Overall, this is a great proposal that addresses some very real pain points.
>
>+1
>
>Thanks,
>Kartikey
>
>On Tue, Sep 23, 2025 at 8:54 AM Yi Zhang <[email protected]> wrote:
>
>> Hi everyone,
>>
>>
>> I would like to start a discussion about FLIP-549: Support Application
>> Management [1].
>>
>>
>> Despite Flink’s widespread adoption, the existing model for running user
>> logic limits observability and execution flexibility, which affects user
>> experience. This FLIP introduces a new application management framework
>> designed to close these gaps and provide a foundation for future
>> improvements.
>>
>>
>> Looking forward to your feedback and suggestions.
>>
>>
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-549%3A+Support+Application+Management
>>
>>
>> Best regards,
>>
>> Yi Zhang
