Re: [DISCUSS] Improve Queryable State and introduce aQueryServerProxy component
Hi yang, +1 for this proposal. Queryable state is a very common usage in our scenarios when we debug and query the realtime status in streaming process like CEP. And we’ve done a lot to improve the “user experience” of this feature like exposing the taskmanager’s proxy port in TaskManagerInfo. I’m looking forward to a more detailed and deeper discussion and I’d like to contribute back to the community on this. Best Regards, Jiayi Liao Original Message Sender:vino yangyanghua1...@gmail.com Recipient:dev@flink.apache.org...@flink.apache.org Date:Friday, Apr 26, 2019 16:41 Subject:Re: [DISCUSS] Improve Queryable State and introduce aQueryServerProxy component Hi Paul, Thanks for your reply. You are right, currently, the queryable state has few users. And I totally agree with you, it makes the streaming works more like a DB. About the architecture and the problem you concern: yes, it maybe affect the JobManager if they are deployed together. I think it's important to guarantee the JobManager's available and stability, and the QueryProxyServer is just a secondary service component. So when describing the role of the QueryProxyServer, I mentioned SLA policy, I think it's a solution. But the detail may need to be discussed. About starting queryable state client with a cmd, I think it's a good idea and valuable. Best, Vino. Paul Lam paullin3...@gmail.com 于2019年4月26日周五 下午3:31写道： Hi Vino, Thanks a lot for bringing up the discussion! Queryable state has been at beta version for a long time, and due to its complexity and instability I think there are not many users, but there’s a great value in it which makes state as database one step closer. WRT the architecture, I’d vote for opt 3, because it fits the cloud architecture the most and avoids putting more burdens on JM (sometimes the queries could be slow and resources intensive). My concern is that on many cluster frameworks the container resources are limited (IIUC, the JM and QS are running in the same container), would JM gets killed if QS eats up too much memory? And a minor suggestion: can we introduce a cmd script to setup a QueryableStateClient? That would be easier for users who wants to try out this feature. Best, Paul Lam在 2019年4月26日，11:09，vino yang yanghua1...@gmail.com 写道： Hi Quan, Thanks for your reply. Actually, I did not try this way. But, there are two factors we should consider: 1. The local state storage is not equals to RocksDB, otherwise Flink does not need to provide a queryable state client. What's more, querying the RocksDB is still an address-explicit action. 2. IMO, the proposal's more valuable suggestion is to make the queryable state's architecture more reasonable, let it encapsulated more details and improve its scalability. Best, Vino Shi Quan qua...@outlook.com 于2019年4月26日周五 上午10:38写道： Hi, How about take states from RocksDB directly, in this case, TM host is unnecessary. Best Quan Shi From: vino yang yanghua1...@gmail.com Sent: Thursday, April 25, 2019 10:18:20 PM To: dev; user Cc: Stefan Richter; Aljoscha Krettek; kklou...@gmail.com Subject: [DISCUSS] Improve Queryable State and introduce a QueryServerProxy component Hi all, I want to share my thought with you about improving the queryable state and introducing a QueryServerProxy component. I think the current queryable state's client is hard to use. Because it needs users to know the TaskManager's address and proxy's port. Actually, some business users who do not have good knowledge about the Flink's inner or runtime in production. However, sometimes they need to query the values of states. IMO, the reason caused this problem is because of the queryable state's architecture. Currently, the queryable state clients interact with query state client proxy components which host on each TaskManager. This design is difficult to encapsulate the point of change and exposes too much detail to the user. My personal idea is that we could introduce a really queryable state server, named e.g. QueryStateProxyServer which would delegate all the query state request and query the local registry then redirect the request to the specific QueryStateClientProxy(runs on each TaskManager). The server is the users really want to care about. And it would make the users ignorant to the TaskManagers' address and proxies' port. The current QueryStateClientProxy would become QueryStateProxyClient. Generally speaking, the roles of the QueryStateProxyServer list below: * works as all the query client's proxy to receive all the request and send response; * a router to redirect the real query requests to the specific proxy client; * maintain route table registry (state - TaskManager, TaskManager-proxy client address) * more fine-grante
Re: [DISCUSS] Improve Queryable State and introduce aQueryServerProxy component
Hi Jiayi, Thanks for your reply and glad to hear that you have taken some effort for it, the potential contribution is also welcome. I also want to explore it in depth. Currently, let's listen to the community's opinions. Best, Vino. bupt_ljy 于2019年4月26日周五 下午9:54写道： > Hi yang, > +1 for this proposal. Queryable state is a very common usage in our > scenarios when we debug and query the realtime status in streaming process > like CEP. And we’ve done a lot to improve the “user experience” of this > feature like exposing the taskmanager’s proxy port in TaskManagerInfo. > I’m looking forward to a more detailed and deeper discussion and I’d like > to contribute back to the community on this. > > > Best Regards, > Jiayi Liao > > > Original Message > Sender:vino yangyanghua1...@gmail.com > Recipient:dev@flink.apache.org...@flink.apache.org > Date:Friday, Apr 26, 2019 16:41 > Subject:Re: [DISCUSS] Improve Queryable State and introduce > aQueryServerProxy component > > > Hi Paul, Thanks for your reply. You are right, currently, the queryable > state has few users. And I totally agree with you, it makes the streaming > works more like a DB. About the architecture and the problem you concern: > yes, it maybe affect the JobManager if they are deployed together. I think > it's important to guarantee the JobManager's available and stability, and > the QueryProxyServer is just a secondary service component. So when > describing the role of the QueryProxyServer, I mentioned SLA policy, I > think it's a solution. But the detail may need to be discussed. About > starting queryable state client with a cmd, I think it's a good idea and > valuable. Best, Vino. Paul Lam paullin3...@gmail.com 于2019年4月26日周五 > 下午3:31写道： Hi Vino, Thanks a lot for bringing up the discussion! > Queryable state has been at beta version for a long time, and due to its > complexity and instability I think there are not many users, but there’s a > great value in it which makes state as database one step closer. WRT the > architecture, I’d vote for opt 3, because it fits the cloud architecture > the most and avoids putting more burdens on JM (sometimes the queries > could be slow and resources intensive). My concern is that on many cluster > frameworks the container resources are limited (IIUC, the JM and QS are > running in the same container), would JM gets killed if QS eats up too > much memory? And a minor suggestion: can we introduce a cmd script to > setup a QueryableStateClient? That would be easier for users who wants to > try out this feature. Best, Paul Lam在 2019年4月26日，11:09，vino yang > yanghua1...@gmail.com 写道： Hi Quan, Thanks for your reply. > Actually, I did not try this way. But, there are two factors we should > consider: 1. The local state storage is not equals to RocksDB, > otherwise Flink does not need to provide a queryable state client. What's > more, querying the RocksDB is still an address-explicit action. 2. > IMO, the proposal's more valuable suggestion is to make the queryable > state's architecture more reasonable, let it encapsulated more details > and improve its scalability. Best, Vino Shi Quan > qua...@outlook.com 于2019年4月26日周五 上午10:38写道： Hi, How about take > states from RocksDB directly, in this case, TM host is unnecessary. > Best Quan Shi From: vino yang > yanghua1...@gmail.com Sent: Thursday, April 25, 2019 10:18:20 PM To: > dev; user Cc: Stefan Richter; Aljoscha Krettek; kklou...@gmail.com > Subject: [DISCUSS] Improve Queryable State and introduce a > QueryServerProxy component Hi all, I want to share my thought with > you about improving the queryable state and introducing a > QueryServerProxy component. I think the current queryable state's > client is hard to use. Because it needs users to know the TaskManager's > address and proxy's port. Actually, some business users who do not have > good knowledge about the Flink's inner or runtime in production. > However, sometimes they need to query the values of states. IMO, the > reason caused this problem is because of the queryable state's > architecture. Currently, the queryable state clients interact with query > state client proxy components which host on each TaskManager. This > design is difficult to encapsulate the point of change and exposes too > much detail to the user. My personal idea is that we could introduce > a really queryable state server, named e.g. QueryStateProxyServer which > would delegate all the query state request and query the local registry > then redirect the request to the specific QueryStateClientProxy(runs on > each TaskManager). The server is the users really want to care about. > And it would make the users ignorant to the TaskManagers' address and > proxies' port. The current QueryStateClientProxy would become > QueryStateProxyClient. Generally spea
Re: RE: [DISCUSS] Improve Queryable State and introduce aQueryServerProxy component
Hi vino, Big +1 for this. Glad to see new progress on this topic! I’ve left some comments on it. Best Regards, Jiayi Liao Original Message Sender:vino yangyanghua1...@gmail.com Recipient:Georgi stoyanovgstoya...@live.com Cc:dev...@flink.apache.org; useru...@flink.apache.org; Stefan richters.rich...@ververica.com; Aljoscha krettekaljos...@apache.org; kkloudas@gmail.comkklou...@gmail.com; Stephan ewense...@apache.org; liyu@apache.orgl...@apache.org; Tzu-Li (Gordon) taitzuli...@apache.org Date:Tuesday, Jul 2, 2019 16:45 Subject:Re: RE: [DISCUSS] Improve Queryable State and introduce aQueryServerProxy component Hi all, In the past, I have tried to further refine the design of this topic thread and wrote a design document to give more detailed design images and text description, so that it is more conducive to discussion.[1] Note: The document is not yet completed, for example, the "Implementation" section is missing. Therefore, it is still in an open discussion state. I will improve the rest while listening to the opinions of the community. Welcome and appreciate more discussions and feedback. Best, Vino [1]:https://docs.google.com/document/d/181qYVIiHQGrc3hCj3QBn1iEHF4bUztdw4XO8VSaf_uI/edit?usp=sharing yanghua1127 yanghua1...@gmail.com 于2019年6月7日周五 下午11:32写道： Hi Georgi, Thanks for your feedback. And glad to hear you are using queryable state. I agree that implementation of option 1 is easier than others. However, when we design the new architecture we need to consider more aspects .e.g. scalability. So it seems option 3 is more suitable. Actually, some committers such as Stefan, Gordon and Aljoscha have given me feedback and direction. Currently, I am writing the design document. If it is ready to be presented. I will copy to this thread and we can discuss further details. Best, Vino On 2019-06-07 19:03 , Georgi Stoyanov Wrote: Hi Vino, I was investigating the current architecture and AFAIK the first proposal will be a lot easier to implement, cause currently JM has the information about the states (where, which etc thanks to KvStateLocationRegistry. Correct me if I’m wrong) We are using the feature and it’s indeed not very cool to iterate trough ports, check which TM is the responsible one etc etc. It will be very useful if someone from the committers joins the topic and give us some insights what’s going to happen with that feature. Kind Regards, Georgi From: vino yang yanghua1...@gmail.com Sent: Thursday, April 25, 2019 5:18 PM To: dev dev@flink.apache.org; user u...@flink.apache.org Cc: Stefan Richter s.rich...@ververica.com; Aljoscha Krettek aljos...@apache.org; kklou...@gmail.com Subject: [DISCUSS] Improve Queryable State and introduce a QueryServerProxy component Hi all, I want to share my thought with you about improving thequeryable state and introducing a QueryServerProxy component. I think the current queryable state's client is hard to use. Because it needs users to know the TaskManager's address and proxy's port. Actually, some business users who do not have good knowledge about the Flink's inner or runtime in production. However, sometimes they need to query the values of states. IMO, the reason caused this problem is because of the queryable state's architecture. Currently, the queryable state clientsinteract with querystate client proxy components which host on each TaskManager.This design is difficult to encapsulate the point of change and exposes too much detail to the user. My personal idea is that we could introduce a really queryable state server, named e.g.QueryStateProxyServerwhich would delegate all the query state request and query the local registry then redirect the request to the specific QueryStateClientProxy(runs on each TaskManager). The server is the users really want to care about. And it would make the users ignorant to the TaskManagers' address and proxies' port. The current QueryStateClientProxy would become QueryStateProxyClient. Generally speaking, the roles of the QueryStateProxyServer list below: works as all the query client's proxy to receive all the request and send response; a router to redirect the real query requests to the specific proxy client; maintain route table registry(state - TaskManager, TaskManager-proxy client address) more fine-granted control, such as cache result, ACL, TTL, SLA(rate limit) and so on About the implementation, there are three opts: opt 1: Let the JobManager acts as the query proxy server. · pros: reuse the exists JM, do not need to introduce a new process can reduce the complexity; · cons: would make JM heavy burdens, depends on the query frequency, may impact on the stability opt 2: Introduce a new component which runs as a single process and acts as the query proxy server: · pros: reduce the burdens and make the JM more stability · cons: introduced a new component will make the implem
Re: RE: [DISCUSS] Improve Queryable State and introduce aQueryServerProxy component
Hi Jiayi, Thanks for your comments. It's valuable. I have accepted it and refined my design document. You can have another review when you have time. Best, Vino bupt_ljy 于2019年7月3日周三 下午2:48写道： > Hi vino, > Big +1 for this. > > Glad to see new progress on this topic! I’ve left some comments on it. > > > Best Regards, > > Jiayi Liao > > Original Message > *Sender:* vino yang > *Recipient:* Georgi Stoyanov > *Cc:* dev; user; Stefan > Richter; Aljoscha Krettek; > kklou...@gmail.com; Stephan Ewen; > l...@apache.org; Tzu-Li (Gordon) Tai > *Date:* Tuesday, Jul 2, 2019 16:45 > *Subject:* Re: RE: [DISCUSS] Improve Queryable State and introduce > aQueryServerProxy component > > Hi all, > > In the past, I have tried to further refine the design of this topic > thread and wrote a design document to give more detailed design images and > text description, so that it is more conducive to discussion.[1] > > Note: The document is not yet completed, for example, the "Implementation" > section is missing. Therefore, it is still in an open discussion state. I > will improve the rest while listening to the opinions of the community. > > Welcome and appreciate more discussions and feedback. > > Best, > Vino > > [1]: > https://docs.google.com/document/d/181qYVIiHQGrc3hCj3QBn1iEHF4bUztdw4XO8VSaf_uI/edit?usp=sharing > > > yanghua1127 于2019年6月7日周五 下午11:32写道： > >> Hi Georgi, >> >> Thanks for your feedback. And glad to hear you are using queryable state. >> >> I agree that implementation of option 1 is easier than others. However, >> when we design the new architecture we need to consider more aspects .e.g. >> scalability. So it seems option 3 is more suitable. Actually, some >> committers such as Stefan, Gordon and Aljoscha have given me feedback and >> direction. >> >> Currently, I am writing the design document. If it is ready to be >> presented. I will copy to this thread and we can discuss further details. >> >> >> Best, >> Vino >> >> >> On 2019-06-07 19:03 , Georgi Stoyanov Wrote: >> >> Hi Vino, >> >> >> >> I was investigating the current architecture and AFAIK the first proposal >> will be a lot easier to implement, cause currently JM has the information >> about the states (where, which etc thanks to KvStateLocationRegistry. >> Correct me if I’m wrong) >> >> We are using the feature and it’s indeed not very cool to iterate trough >> ports, check which TM is the responsible one etc etc. >> >> >> >> It will be very useful if someone from the committers joins the topic and >> give us some insights what’s going to happen with that feature. >> >> >> >> >> >> Kind Regards, >> >> Georgi >> >> >> >> >> >> >> >> *From:* vino yang >> *Sent:* Thursday, April 25, 2019 5:18 PM >> *To:* dev ; user >> *Cc:* Stefan Richter ; Aljoscha Krettek < >> aljos...@apache.org>; kklou...@gmail.com >> *Subject:* [DISCUSS] Improve Queryable State and introduce a >> QueryServerProxy component >> >> >> >> Hi all, >> >> >> >> I want to share my thought with you about improving the queryable state >> and introducing a QueryServerProxy component. >> >> >> >> I think the current queryable state's client is hard to use. Because it >> needs users to know the TaskManager's address and proxy's port. Actually, >> some business users who do not have good knowledge about the Flink's inner >> or runtime in production. However, sometimes they need to query the values >> of states. >> >> >> >> IMO, the reason caused this problem is because of the queryable state's >> architecture. Currently, the queryable state clients interact with >> query state client proxy components which host on each TaskManager. This >> design is difficult to encapsulate the point of change and exposes too much >> detail to the user. >> >> >> >> My personal idea is that we could introduce a really queryable state >> server, named e.g. *QueryStateProxyServer* which would delegate all the >> query state request and query the local registry then redirect the request >> to the specific *QueryStateClientProxy*(runs on each TaskManager). The >> server is the users really want to care about. And it would make the users >> ignorant to the TaskManagers' address and proxies' port. The current >> *QueryStateClientProxy* would become *QueryStateProxyClient*. >
Re: RE: [DISCUSS] Improve Queryable State and introduce aQueryServerProxy component
Hi Vino, Thanks for the write up. I've left some comments on google doc, please check. Thanks. Best Regards, Yu On Thu, 4 Jul 2019 at 15:37, vino yang wrote: > Hi Jiayi, > > Thanks for your comments. > > It's valuable. I have accepted it and refined my design document. > > You can have another review when you have time. > > Best, > Vino > > bupt_ljy 于2019年7月3日周三 下午2:48写道： > > > Hi vino, > > Big +1 for this. > > > > Glad to see new progress on this topic! I’ve left some comments on it. > > > > > > Best Regards, > > > > Jiayi Liao > > > > Original Message > > *Sender:* vino yang > > *Recipient:* Georgi Stoyanov > > *Cc:* dev; user; Stefan > > Richter; Aljoscha Krettek; > > kklou...@gmail.com; Stephan Ewen; > > l...@apache.org; Tzu-Li (Gordon) Tai< > tzuli...@apache.org> > > *Date:* Tuesday, Jul 2, 2019 16:45 > > *Subject:* Re: RE: [DISCUSS] Improve Queryable State and introduce > > aQueryServerProxy component > > > > Hi all, > > > > In the past, I have tried to further refine the design of this topic > > thread and wrote a design document to give more detailed design images > and > > text description, so that it is more conducive to discussion.[1] > > > > Note: The document is not yet completed, for example, the > "Implementation" > > section is missing. Therefore, it is still in an open discussion state. I > > will improve the rest while listening to the opinions of the community. > > > > Welcome and appreciate more discussions and feedback. > > > > Best, > > Vino > > > > [1]: > > > https://docs.google.com/document/d/181qYVIiHQGrc3hCj3QBn1iEHF4bUztdw4XO8VSaf_uI/edit?usp=sharing > > > > > > yanghua1127 于2019年6月7日周五 下午11:32写道： > > > >> Hi Georgi, > >> > >> Thanks for your feedback. And glad to hear you are using queryable > state. > >> > >> I agree that implementation of option 1 is easier than others. However, > >> when we design the new architecture we need to consider more aspects > .e.g. > >> scalability. So it seems option 3 is more suitable. Actually, some > >> committers such as Stefan, Gordon and Aljoscha have given me feedback > and > >> direction. > >> > >> Currently, I am writing the design document. If it is ready to be > >> presented. I will copy to this thread and we can discuss further > details. > >> > >> > >> Best, > >> Vino > >> > >> > >> On 2019-06-07 19:03 , Georgi Stoyanov Wrote: > >> > >> Hi Vino, > >> > >> > >> > >> I was investigating the current architecture and AFAIK the first > proposal > >> will be a lot easier to implement, cause currently JM has the > information > >> about the states (where, which etc thanks to KvStateLocationRegistry. > >> Correct me if I’m wrong) > >> > >> We are using the feature and it’s indeed not very cool to iterate trough > >> ports, check which TM is the responsible one etc etc. > >> > >> > >> > >> It will be very useful if someone from the committers joins the topic > and > >> give us some insights what’s going to happen with that feature. > >> > >> > >> > >> > >> > >> Kind Regards, > >> > >> Georgi > >> > >> > >> > >> > >> > >> > >> > >> *From:* vino yang > >> *Sent:* Thursday, April 25, 2019 5:18 PM > >> *To:* dev ; user > >> *Cc:* Stefan Richter ; Aljoscha Krettek < > >> aljos...@apache.org>; kklou...@gmail.com > >> *Subject:* [DISCUSS] Improve Queryable State and introduce a > >> QueryServerProxy component > >> > >> > >> > >> Hi all, > >> > >> > >> > >> I want to share my thought with you about improving the queryable state > >> and introducing a QueryServerProxy component. > >> > >> > >> > >> I think the current queryable state's client is hard to use. Because it > >> needs users to know the TaskManager's address and proxy's port. > Actually, > >> some business users who do not have good knowledge about the Flink's > inner > >> or runtime in production. However, sometimes they need to query the > values > >> of states. > >> > >> > >> > >> IMO, the r
