Thanks for your clarification Till. For terminology pluggable and customizable, I am mainly concerning about interface audience issue. Pluggable means we have multiple high-availability implementation but closed to extend in user scope; customizable means high-availability interface are stable and user-facing.
I agree that we should try to keep backward compatibility if possible. I start this thread due to being confused how I progress FLINK-10333. Specifically, how to deal with compatibility things. Given that it is an internal interface I think it is reasonable we evolve it for supporting leader store based high-availability storage in a minor version bump. I'm going to start a discuss thread next week for gathering wider feedbacks beyond the original (maybe limited) JIRA, which also calls of review among community members. What do you think? Best, tison. Till Rohrmann <trohrm...@apache.org> 于2019年10月21日周一 下午5:21写道： > Hi Tison, > > I'm not sure whether I fully understand your distinction between > customizable and pluggable. Maybe you could clarify your ideas a bit > because you seem to favour support for pluggable implementations. > > Maybe let me try to answer some other questions you raised. With the > HighAvailabilityServices interface and the functionality to load custom > implementations specified via `high-availability: FQDN` it is indeed > possible to provide a custom implementation of the HA services. This is, > however, pretty much a power user feature where users have to implement > against an internal API. As such, it can be subject to change and we don't > give guarantees that these interfaces won't change. Of course, if possible, > we should extend/change it in a way that we guarantee backwards > compatibility. > > You are right that at the moment this interface is not stable enough for > being public API. Once this changes and we are happy with it, then we can > think about making it a public API and documenting it properly. Afaik, > there is no documentation how to implement your own HA services at the > moment. This underlines as well that this interface is an internal API. > > Cheers, > Till > > On Fri, Oct 18, 2019 at 5:56 AM Zili Chen <wander4...@gmail.com> wrote: > > > Another perspective is that a stable, carefully-designed interface with > > clear semantic > > could be safer to customize. > > > > Following the discussion in FLINK-10333 our JobGraphStore is actually > > required > > performing write operation only with leadership, which is a basic > > requirement > > for coordination rather than an implementation detail. > > Thus it depends on LeaderElectionService(in the design, we narrow the > > specific interface > > as LeaderStore). HighAvailabilityServices#getJobGraphStore() infers a > > implicit field for > > that which is hard to express the relationship between them. > > > > If the interface is unstable(also we introduce a ClientHAService for > > separate concern and > > have to keep b/w comp. for customized), we'd better keep it internal for > > freely evolution. > > And when we try to support customized, it would be helpful to start a > > proposal to revisit > > the interface to be well-designed and stable. ref[2]. > > > > In short, current high-availability services as well as > > runtime/coordination is still under > > development and active evolution. It is possibly not a good time for make > > it public and > > customizable. > > > > Best, > > tison. > > > > [1] https://issues.apache.org/jira/browse/FLINK-10333 > > [2] > > > > > https://github.com/apache/pulsar/wiki/PIP-45%3A-Pluggable-metadata-interface > > > > > > Zili Chen <wander4...@gmail.com> 于2019年10月17日周四 下午8:13写道： > > > > > A challenge is how we ensure the support for customized implementation. > > > When > > > we introduce JobGraphStore#releaseJobGraph we actually change quite a > bit > > > codepath > > > in Dispatcher. While we are unable to test arbitrarily customized > > > implementation our > > > compatibility promise is actually no more than compilation compatible. > > > > > > Customer should still be required to be familiar with implementation > > > details to figure > > > out the fitment when they bump Flink version. This effort requires also > > > and no extra > > > when we support pluggable strategy. In another word, a customized > support > > > tends > > > to hide the challenge when customer want to use their own > implementation. > > > > > >
