Hi yang,
+1 for this proposal. Queryable state is a very common usage in our scenarios
when we debug and query the realtime status in streaming process like CEP. And
we’ve done a lot to improve the “user experience” of this feature like exposing
the taskmanager’s proxy port in TaskManagerInfo.
I’m looking forward to a more detailed and deeper discussion and I’d like to
contribute back to the community on this.
Best Regards,
Jiayi Liao
Original Message
Sender:vino yangyanghua1...@gmail.com
Recipient:dev@flink.apache.org...@flink.apache.org
Date:Friday, Apr 26, 2019 16:41
Subject:Re: [DISCUSS] Improve Queryable State and introduce aQueryServerProxy
component
Hi Paul, Thanks for your reply. You are right, currently, the queryable state
has few users. And I totally agree with you, it makes the streaming works more
like a DB. About the architecture and the problem you concern: yes, it maybe
affect the JobManager if they are deployed together. I think it's important to
guarantee the JobManager's available and stability, and the QueryProxyServer is
just a secondary service component. So when describing the role of the
QueryProxyServer, I mentioned SLA policy, I think it's a solution. But the
detail may need to be discussed. About starting queryable state client with a
cmd, I think it's a good idea and valuable. Best, Vino. Paul Lam
paullin3...@gmail.com 于2019年4月26日周五 下午3:31写道： Hi Vino, Thanks a lot for
bringing up the discussion! Queryable state has been at beta version for a
long time, and due to its complexity and instability I think there are not
many users, but there’s a great value in it which makes state as database one
step closer. WRT the architecture, I’d vote for opt 3, because it fits the
cloud architecture the most and avoids putting more burdens on JM (sometimes
the queries could be slow and resources intensive). My concern is that on many
cluster frameworks the container resources are limited (IIUC, the JM and QS
are running in the same container), would JM gets killed if QS eats up too
much memory? And a minor suggestion: can we introduce a cmd script to setup a
QueryableStateClient? That would be easier for users who wants to try out
this feature. Best, Paul Lam 在 2019年4月26日，11:09，vino yang
yanghua1...@gmail.com 写道： Hi Quan, Thanks for your reply. Actually,
I did not try this way. But, there are two factors we should consider:
1. The local state storage is not equals to RocksDB, otherwise Flink does
not need to provide a queryable state client. What's more, querying the
RocksDB is still an address-explicit action. 2. IMO, the proposal's more
valuable suggestion is to make the queryable state's architecture more
reasonable, let it encapsulated more details and improve its scalability.
Best, Vino Shi Quan qua...@outlook.com 于2019年4月26日周五 上午10:38写道：
Hi, How about take states from RocksDB directly, in this case, TM host is
unnecessary. Best Quan Shi ________________________________ From:
vino yang yanghua1...@gmail.com Sent: Thursday, April 25, 2019 10:18:20 PM
To: dev; user Cc: Stefan Richter; Aljoscha Krettek; kklou...@gmail.com
Subject: [DISCUSS] Improve Queryable State and introduce a QueryServerProxy
component Hi all, I want to share my thought with you about improving
the queryable state and introducing a QueryServerProxy component. I think
the current queryable state's client is hard to use. Because it needs users
to know the TaskManager's address and proxy's port. Actually, some business
users who do not have good knowledge about the Flink's inner or runtime in
production. However, sometimes they need to query the values of states.
IMO, the reason caused this problem is because of the queryable state's
architecture. Currently, the queryable state clients interact with query
state client proxy components which host on each TaskManager. This design is
difficult to encapsulate the point of change and exposes too much detail to
the user. My personal idea is that we could introduce a really queryable
state server, named e.g. QueryStateProxyServer which would delegate all the
query state request and query the local registry then redirect the request to
the specific QueryStateClientProxy(runs on each TaskManager). The server is
the users really want to care about. And it would make the users ignorant to
the TaskManagers' address and proxies' port. The current
QueryStateClientProxy would become QueryStateProxyClient. Generally
speaking, the roles of the QueryStateProxyServer list below: * works as
all the query client's proxy to receive all the request and send response;
* a router to redirect the real query requests to the specific proxy client;
* maintain route table registry (state - TaskManager, TaskManager-proxy
client address) * more fine-granted control, such as cache result, ACL, TTL,
SLA(rate limit) and so on About the implementation, there are three opts:
opt 1: Let the JobManager acts as the query proxy server. * pros:
reuse the exists JM, do not need to introduce a new process can reduce the
complexity; * cons: would make JM heavy burdens, depends on the query
frequency, may impact on the stability [Screen Shot 2019-04-25 at 5.12.07
PM.png] opt 2: Introduce a new component which runs as a single process
and acts as the query proxy server: * pros: reduce the burdens and
make the JM more stability * cons: introduced a new component will make the
implementation more complexity [Screen Shot 2019-04-25 at 5.14.05 PM.png]
opt 3 (suggestion comes from Stefan Richter): Combining the two opts,
the query server could run as a single entry point(process) and integrate
with JobManager. If we keep it well encapsulated, the only difference would
be how we register new TMs with the query server in the different scenarios,
in JM we might have this information already, in standalone e.g. the TMs be
started with the query server address to register. This would give the
convenience to start QS with the JM and the flexibility for power user to
reduce load on their JM. IMO, the queryable state is a very valuable
feature. It can let users query some real-time measure results. I hope it
will get the attention of the community. It is just a roughly thought.
If it is valuable to the community, I will give a design draft. What's
your opinion? Any feedback and comment are welcome! Best, Vino.
