[GitHub] flink pull request #4966: [FLINK-7822][FLINK-7823] Adds documentation and fi...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/4966 ---
[GitHub] flink pull request #4966: [FLINK-7822][FLINK-7823] Adds documentation and fi...
Github user kl0u commented on a diff in the pull request:
https://github.com/apache/flink/pull/4966#discussion_r149340569
--- Diff: docs/dev/stream/state/queryable_state.md ---
@@ -32,38 +32,67 @@ under the License.
likely that there will be breaking API changes on the client side in the
upcoming Flink versions.
-In a nutshell, this feature allows users to query Flink's managed
partitioned state
-(see [Working with State]({{ site.baseurl }}/dev/stream/state/state.html))
from outside of
-Flink. For some scenarios, queryable state thus eliminates the need for
distributed
-operations/transactions with external systems such as key-value stores
which are often the
-bottleneck in practice.
+In a nutshell, this feature exposes Flink's managed keyed (partitioned)
state
+(see [Working with State]({{ site.baseurl }}/dev/stream/state/state.html))
to the outside world and
+allows the user to query a job's state from outside Flink. For some
scenarios, queryable state
+eliminates the need for distributed operations/transactions with external
systems such as key-value
+stores which are often the bottleneck in practice. In addition, this
feature may be particularly
+useful for debugging purposes.
- Attention: Queryable state accesses keyed state from a
concurrent thread rather
- than synchronizing with the operator and potentially blocking its
operation. Since any state
- backend using Java heap space, e.g. MemoryStateBackend or
- FsStateBackend, does not work with copies when retrieving values but
instead directly
- references the stored values, read-modify-write patterns are unsafe and
may cause the
- queryable state server to fail due to concurrent modifications.
- The RocksDBStateBackend is safe from these issues.
+ Attention: When querying a state object, that object is
accessed from a concurrent
+ thread without any synchronization or copying. This is a design choice,
as any of the above would lead
+ to increased job latency, which we wanted to avoid. Since any state
backend using Java heap space,
+ e.g. MemoryStateBackend or
FsStateBackend, does not work
+ with copies when retrieving values but instead directly references the
stored values, read-modify-write
+ patterns are unsafe and may cause the queryable state server to fail due
to concurrent modifications.
+ The RocksDBStateBackend is safe from these issues.
+## Architecture
+
+Before showing how to use the Queryable State, it is useful to briefly
describe the entities that compose it.
+The Queryable State consists of three main entities:
+
+ 1. the `QueryableStateClient`, which (potentially) runs outside the Flink
cluster and submits the user queries,
+ 2. the `QueryableStateClientProxy`, which runs on each `TaskManager`
(*i.e.* inside the Flink cluster) and is responsible
+ for receiving the client's queries, fetching the requested state on his
behalf, and returning it to the client, and
+ 3. the `QueryableStateServer` which runs on each `TaskManager` and is
responsible for serving the locally stored state.
+
+In a nutshell, the client will connect to one of the proxies and send a
request for the state associated with a specific
+key, `k`. As stated in [Working with State]({{ site.baseurl
}}/dev/stream/state/state.html), keyed state is organized in
+*Key Groups*, and each `TaskManager` is assigned a number of these key
groups. To discover which `TaskManager` is
+responsible for the key group holding `k`, the proxy will ask the
`JobManager`. Based on the answer, the proxy will
+then query the `QueryableStateServer` running on that `TaskManager` for
the state associated with `k`, and forward the
+response back to the client.
+
+## Activating Queryable State
+
+To enable queryable state on your Flink cluster, you just have to copy the
--- End diff --
You mean make it: "To enable **the** queryable state on your Flink
cluster..."?
---
[GitHub] flink pull request #4966: [FLINK-7822][FLINK-7823] Adds documentation and fi...
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/4966#discussion_r149337397
--- Diff: docs/dev/stream/state/queryable_state.md ---
@@ -32,38 +32,67 @@ under the License.
likely that there will be breaking API changes on the client side in the
upcoming Flink versions.
-In a nutshell, this feature allows users to query Flink's managed
partitioned state
-(see [Working with State]({{ site.baseurl }}/dev/stream/state/state.html))
from outside of
-Flink. For some scenarios, queryable state thus eliminates the need for
distributed
-operations/transactions with external systems such as key-value stores
which are often the
-bottleneck in practice.
+In a nutshell, this feature exposes Flink's managed keyed (partitioned)
state
+(see [Working with State]({{ site.baseurl }}/dev/stream/state/state.html))
to the outside world and
+allows the user to query a job's state from outside Flink. For some
scenarios, queryable state
+eliminates the need for distributed operations/transactions with external
systems such as key-value
+stores which are often the bottleneck in practice. In addition, this
feature may be particularly
+useful for debugging purposes.
- Attention: Queryable state accesses keyed state from a
concurrent thread rather
- than synchronizing with the operator and potentially blocking its
operation. Since any state
- backend using Java heap space, e.g. MemoryStateBackend or
- FsStateBackend, does not work with copies when retrieving values but
instead directly
- references the stored values, read-modify-write patterns are unsafe and
may cause the
- queryable state server to fail due to concurrent modifications.
- The RocksDBStateBackend is safe from these issues.
+ Attention: When querying a state object, that object is
accessed from a concurrent
+ thread without any synchronization or copying. This is a design choice,
as any of the above would lead
+ to increased job latency, which we wanted to avoid. Since any state
backend using Java heap space,
+ e.g. MemoryStateBackend or
FsStateBackend, does not work
+ with copies when retrieving values but instead directly references the
stored values, read-modify-write
+ patterns are unsafe and may cause the queryable state server to fail due
to concurrent modifications.
+ The RocksDBStateBackend is safe from these issues.
+## Architecture
+
+Before showing how to use the Queryable State, it is useful to briefly
describe the entities that compose it.
+The Queryable State consists of three main entities:
+
+ 1. the `QueryableStateClient`, which (potentially) runs outside the Flink
cluster and submits the user queries,
+ 2. the `QueryableStateClientProxy`, which runs on each `TaskManager`
(*i.e.* inside the Flink cluster) and is responsible
+ for receiving the client's queries, fetching the requested state on his
behalf, and returning it to the client, and
--- End diff --
maybe "fetching the requested state from the responsible TaskManager"
---
[GitHub] flink pull request #4966: [FLINK-7822][FLINK-7823] Adds documentation and fi...
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/4966#discussion_r149337715
--- Diff: docs/dev/stream/state/queryable_state.md ---
@@ -32,38 +32,67 @@ under the License.
likely that there will be breaking API changes on the client side in the
upcoming Flink versions.
-In a nutshell, this feature allows users to query Flink's managed
partitioned state
-(see [Working with State]({{ site.baseurl }}/dev/stream/state/state.html))
from outside of
-Flink. For some scenarios, queryable state thus eliminates the need for
distributed
-operations/transactions with external systems such as key-value stores
which are often the
-bottleneck in practice.
+In a nutshell, this feature exposes Flink's managed keyed (partitioned)
state
+(see [Working with State]({{ site.baseurl }}/dev/stream/state/state.html))
to the outside world and
+allows the user to query a job's state from outside Flink. For some
scenarios, queryable state
+eliminates the need for distributed operations/transactions with external
systems such as key-value
+stores which are often the bottleneck in practice. In addition, this
feature may be particularly
+useful for debugging purposes.
- Attention: Queryable state accesses keyed state from a
concurrent thread rather
- than synchronizing with the operator and potentially blocking its
operation. Since any state
- backend using Java heap space, e.g. MemoryStateBackend or
- FsStateBackend, does not work with copies when retrieving values but
instead directly
- references the stored values, read-modify-write patterns are unsafe and
may cause the
- queryable state server to fail due to concurrent modifications.
- The RocksDBStateBackend is safe from these issues.
+ Attention: When querying a state object, that object is
accessed from a concurrent
+ thread without any synchronization or copying. This is a design choice,
as any of the above would lead
+ to increased job latency, which we wanted to avoid. Since any state
backend using Java heap space,
+ e.g. MemoryStateBackend or
FsStateBackend, does not work
+ with copies when retrieving values but instead directly references the
stored values, read-modify-write
+ patterns are unsafe and may cause the queryable state server to fail due
to concurrent modifications.
+ The RocksDBStateBackend is safe from these issues.
+## Architecture
+
+Before showing how to use the Queryable State, it is useful to briefly
describe the entities that compose it.
+The Queryable State consists of three main entities:
+
+ 1. the `QueryableStateClient`, which (potentially) runs outside the Flink
cluster and submits the user queries,
+ 2. the `QueryableStateClientProxy`, which runs on each `TaskManager`
(*i.e.* inside the Flink cluster) and is responsible
+ for receiving the client's queries, fetching the requested state on his
behalf, and returning it to the client, and
+ 3. the `QueryableStateServer` which runs on each `TaskManager` and is
responsible for serving the locally stored state.
+
+In a nutshell, the client will connect to one of the proxies and send a
request for the state associated with a specific
+key, `k`. As stated in [Working with State]({{ site.baseurl
}}/dev/stream/state/state.html), keyed state is organized in
+*Key Groups*, and each `TaskManager` is assigned a number of these key
groups. To discover which `TaskManager` is
+responsible for the key group holding `k`, the proxy will ask the
`JobManager`. Based on the answer, the proxy will
+then query the `QueryableStateServer` running on that `TaskManager` for
the state associated with `k`, and forward the
+response back to the client.
+
+## Activating Queryable State
+
+To enable queryable state on your Flink cluster, you just have to copy the
+`flink-queryable-state-runtime{{ site.scala_version_suffix
}}-{{site.version }}.jar`
+from the `opt/` folder of your [Flink
distribution](https://flink.apache.org/downloads.html "Apache Flink:
Downloads"),
+to the `lib/` folder. In other case, the queryable state feature is not
enabled.
--- End diff --
nit: "Otherwise, ..."
---
[GitHub] flink pull request #4966: [FLINK-7822][FLINK-7823] Adds documentation and fi...
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/4966#discussion_r149337260
--- Diff: docs/dev/stream/state/queryable_state.md ---
@@ -32,38 +32,67 @@ under the License.
likely that there will be breaking API changes on the client side in the
upcoming Flink versions.
-In a nutshell, this feature allows users to query Flink's managed
partitioned state
-(see [Working with State]({{ site.baseurl }}/dev/stream/state/state.html))
from outside of
-Flink. For some scenarios, queryable state thus eliminates the need for
distributed
-operations/transactions with external systems such as key-value stores
which are often the
-bottleneck in practice.
+In a nutshell, this feature exposes Flink's managed keyed (partitioned)
state
+(see [Working with State]({{ site.baseurl }}/dev/stream/state/state.html))
to the outside world and
+allows the user to query a job's state from outside Flink. For some
scenarios, queryable state
+eliminates the need for distributed operations/transactions with external
systems such as key-value
+stores which are often the bottleneck in practice. In addition, this
feature may be particularly
+useful for debugging purposes.
- Attention: Queryable state accesses keyed state from a
concurrent thread rather
- than synchronizing with the operator and potentially blocking its
operation. Since any state
- backend using Java heap space, e.g. MemoryStateBackend or
- FsStateBackend, does not work with copies when retrieving values but
instead directly
- references the stored values, read-modify-write patterns are unsafe and
may cause the
- queryable state server to fail due to concurrent modifications.
- The RocksDBStateBackend is safe from these issues.
+ Attention: When querying a state object, that object is
accessed from a concurrent
+ thread without any synchronization or copying. This is a design choice,
as any of the above would lead
+ to increased job latency, which we wanted to avoid. Since any state
backend using Java heap space,
+ e.g. MemoryStateBackend or
FsStateBackend, does not work
+ with copies when retrieving values but instead directly references the
stored values, read-modify-write
+ patterns are unsafe and may cause the queryable state server to fail due
to concurrent modifications.
+ The RocksDBStateBackend is safe from these issues.
+## Architecture
+
+Before showing how to use the Queryable State, it is useful to briefly
describe the entities that compose it.
+The Queryable State consists of three main entities:
--- End diff --
nit: "The Queryable State feature"
---
[GitHub] flink pull request #4966: [FLINK-7822][FLINK-7823] Adds documentation and fi...
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/4966#discussion_r149337820
--- Diff: docs/dev/stream/state/queryable_state.md ---
@@ -32,38 +32,67 @@ under the License.
likely that there will be breaking API changes on the client side in the
upcoming Flink versions.
-In a nutshell, this feature allows users to query Flink's managed
partitioned state
-(see [Working with State]({{ site.baseurl }}/dev/stream/state/state.html))
from outside of
-Flink. For some scenarios, queryable state thus eliminates the need for
distributed
-operations/transactions with external systems such as key-value stores
which are often the
-bottleneck in practice.
+In a nutshell, this feature exposes Flink's managed keyed (partitioned)
state
+(see [Working with State]({{ site.baseurl }}/dev/stream/state/state.html))
to the outside world and
+allows the user to query a job's state from outside Flink. For some
scenarios, queryable state
+eliminates the need for distributed operations/transactions with external
systems such as key-value
+stores which are often the bottleneck in practice. In addition, this
feature may be particularly
+useful for debugging purposes.
- Attention: Queryable state accesses keyed state from a
concurrent thread rather
- than synchronizing with the operator and potentially blocking its
operation. Since any state
- backend using Java heap space, e.g. MemoryStateBackend or
- FsStateBackend, does not work with copies when retrieving values but
instead directly
- references the stored values, read-modify-write patterns are unsafe and
may cause the
- queryable state server to fail due to concurrent modifications.
- The RocksDBStateBackend is safe from these issues.
+ Attention: When querying a state object, that object is
accessed from a concurrent
+ thread without any synchronization or copying. This is a design choice,
as any of the above would lead
+ to increased job latency, which we wanted to avoid. Since any state
backend using Java heap space,
+ e.g. MemoryStateBackend or
FsStateBackend, does not work
+ with copies when retrieving values but instead directly references the
stored values, read-modify-write
+ patterns are unsafe and may cause the queryable state server to fail due
to concurrent modifications.
+ The RocksDBStateBackend is safe from these issues.
+## Architecture
+
+Before showing how to use the Queryable State, it is useful to briefly
describe the entities that compose it.
+The Queryable State consists of three main entities:
+
+ 1. the `QueryableStateClient`, which (potentially) runs outside the Flink
cluster and submits the user queries,
+ 2. the `QueryableStateClientProxy`, which runs on each `TaskManager`
(*i.e.* inside the Flink cluster) and is responsible
+ for receiving the client's queries, fetching the requested state on his
behalf, and returning it to the client, and
+ 3. the `QueryableStateServer` which runs on each `TaskManager` and is
responsible for serving the locally stored state.
+
+In a nutshell, the client will connect to one of the proxies and send a
request for the state associated with a specific
+key, `k`. As stated in [Working with State]({{ site.baseurl
}}/dev/stream/state/state.html), keyed state is organized in
+*Key Groups*, and each `TaskManager` is assigned a number of these key
groups. To discover which `TaskManager` is
+responsible for the key group holding `k`, the proxy will ask the
`JobManager`. Based on the answer, the proxy will
+then query the `QueryableStateServer` running on that `TaskManager` for
the state associated with `k`, and forward the
+response back to the client.
+
+## Activating Queryable State
+
+To enable queryable state on your Flink cluster, you just have to copy the
+`flink-queryable-state-runtime{{ site.scala_version_suffix
}}-{{site.version }}.jar`
+from the `opt/` folder of your [Flink
distribution](https://flink.apache.org/downloads.html "Apache Flink:
Downloads"),
+to the `lib/` folder. In other case, the queryable state feature is not
enabled.
+
+To verify that your cluster is running with queryable state enabled, check
the logs of any
+task manager for the line: `"Started the Queryable State Proxy Server @
..."`.
+
## Making State Queryable
-In order to make state queryable, the queryable state server first needs
to be enabled globally
-by setting the `query.server.enable` configuration parameter to `true`
(current default).
-Then appropriate state needs to be made queryable by using either
+Now that you have activated queryable state on your cluster, it is time to
see how to use it. In order for a state to
--- End d
[GitHub] flink pull request #4966: [FLINK-7822][FLINK-7823] Adds documentation and fi...
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/4966#discussion_r149337649
--- Diff: docs/dev/stream/state/queryable_state.md ---
@@ -32,38 +32,67 @@ under the License.
likely that there will be breaking API changes on the client side in the
upcoming Flink versions.
-In a nutshell, this feature allows users to query Flink's managed
partitioned state
-(see [Working with State]({{ site.baseurl }}/dev/stream/state/state.html))
from outside of
-Flink. For some scenarios, queryable state thus eliminates the need for
distributed
-operations/transactions with external systems such as key-value stores
which are often the
-bottleneck in practice.
+In a nutshell, this feature exposes Flink's managed keyed (partitioned)
state
+(see [Working with State]({{ site.baseurl }}/dev/stream/state/state.html))
to the outside world and
+allows the user to query a job's state from outside Flink. For some
scenarios, queryable state
+eliminates the need for distributed operations/transactions with external
systems such as key-value
+stores which are often the bottleneck in practice. In addition, this
feature may be particularly
+useful for debugging purposes.
- Attention: Queryable state accesses keyed state from a
concurrent thread rather
- than synchronizing with the operator and potentially blocking its
operation. Since any state
- backend using Java heap space, e.g. MemoryStateBackend or
- FsStateBackend, does not work with copies when retrieving values but
instead directly
- references the stored values, read-modify-write patterns are unsafe and
may cause the
- queryable state server to fail due to concurrent modifications.
- The RocksDBStateBackend is safe from these issues.
+ Attention: When querying a state object, that object is
accessed from a concurrent
+ thread without any synchronization or copying. This is a design choice,
as any of the above would lead
+ to increased job latency, which we wanted to avoid. Since any state
backend using Java heap space,
+ e.g. MemoryStateBackend or
FsStateBackend, does not work
+ with copies when retrieving values but instead directly references the
stored values, read-modify-write
+ patterns are unsafe and may cause the queryable state server to fail due
to concurrent modifications.
+ The RocksDBStateBackend is safe from these issues.
+## Architecture
+
+Before showing how to use the Queryable State, it is useful to briefly
describe the entities that compose it.
+The Queryable State consists of three main entities:
+
+ 1. the `QueryableStateClient`, which (potentially) runs outside the Flink
cluster and submits the user queries,
+ 2. the `QueryableStateClientProxy`, which runs on each `TaskManager`
(*i.e.* inside the Flink cluster) and is responsible
+ for receiving the client's queries, fetching the requested state on his
behalf, and returning it to the client, and
+ 3. the `QueryableStateServer` which runs on each `TaskManager` and is
responsible for serving the locally stored state.
+
+In a nutshell, the client will connect to one of the proxies and send a
request for the state associated with a specific
+key, `k`. As stated in [Working with State]({{ site.baseurl
}}/dev/stream/state/state.html), keyed state is organized in
+*Key Groups*, and each `TaskManager` is assigned a number of these key
groups. To discover which `TaskManager` is
+responsible for the key group holding `k`, the proxy will ask the
`JobManager`. Based on the answer, the proxy will
+then query the `QueryableStateServer` running on that `TaskManager` for
the state associated with `k`, and forward the
+response back to the client.
+
+## Activating Queryable State
+
+To enable queryable state on your Flink cluster, you just have to copy the
--- End diff --
nit: omit "the"
---
[GitHub] flink pull request #4966: [FLINK-7822][FLINK-7823] Adds documentation and fi...
GitHub user kl0u opened a pull request: https://github.com/apache/flink/pull/4966 [FLINK-7822][FLINK-7823] Adds documentation and fixes configuration of QS. ## What is the purpose of the change *This PR adds documentation for the new queryable state. R @aljoscha You can merge this pull request into a Git repository by running: $ git pull https://github.com/kl0u/flink qs-doc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/4966.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4966 commit 91c7082158e534ffc30894e9f23e6ea6c94fab07 Author: kkloudas Date: 2017-11-06T11:43:18Z [FLINK-7823][QS] Update Queryable State configuration parameters. commit 2a076d6119eb01a4b4bc48925a37dde1b439dd54 Author: kkloudas Date: 2017-11-06T16:21:45Z [FLINK-7822][QS][doc] Update Queryable State docs. ---
