[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[
https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574882#comment-16574882
]
ASF GitHub Bot commented on FLINK-9637:
---
asfgit closed pull request #6379: [FLINK-9637] Add public user documentation
for state TTL feature
URL: https://github.com/apache/flink/pull/6379
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/docs/dev/stream/state/state.md b/docs/dev/stream/state/state.md
index 44a3653a61f..72fca3e27f1 100644
--- a/docs/dev/stream/state/state.md
+++ b/docs/dev/stream/state/state.md
@@ -266,6 +266,132 @@ a `ValueState`. Once the count reaches 2 it will emit the
average and clear the
we start over from `0`. Note that this would keep a different state value for
each different input
key if we had tuples with different values in the first field.
+### State time-to-live (TTL)
+
+A time-to-live (TTL) can be assigned to the keyed state of any type.
+In this case it will expire after the configured TTL
+and its stored value will be cleaned up on the best effort basis which is
discussed in details later.
+
+The state collection types support per-entry TTLs: list elements and map
entries expire independently.
+
+To use state TTL you must first build a `StateTtlConfig` object,
+then TTL functionality can be enabled in any state descriptor passing this
configuration:
+
+
+
+{% highlight java %}
+import org.apache.flink.api.common.state.StateTtlConfig;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+
+StateTtlConfig ttlConfig = StateTtlConfig
+.newBuilder(Time.seconds(1))
+.setUpdateType(StateTtlConfig.UpdateType.OnCreateAndWrite)
+.setStateVisibility(StateTtlConfig.StateVisibility.NeverReturnExpired)
+.build();
+
+ValueStateDescriptor stateDescriptor = new
ValueStateDescriptor<>("text state", String.class);
+stateDescriptor.enableTimeToLive(ttlConfig);
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+import org.apache.flink.api.common.state.StateTtlConfig
+import org.apache.flink.api.common.time.Time
+import org.apache.flink.api.common.state.ValueStateDescriptor
+
+val ttlConfig = StateTtlConfig
+.newBuilder(Time.seconds(1))
+.setUpdateType(StateTtlConfig.UpdateType.OnCreateAndWrite)
+.setStateVisibility(StateTtlConfig.StateVisibility.NeverReturnExpired)
+.build
+
+val stateDescriptor = new ValueStateDescriptor[String]("text state",
classOf[String])
+stateDescriptor.enableTimeToLive(ttlConfig)
+{% endhighlight %}
+
+
+
+The configuration has several options to consider.
+The first parameter of `newBuilder` method is mandatory, it is the
time-to-live value.
+
+The update type configures when the state TTL is refreshed (default
`OnCreateAndWrite`):
+
+ - `StateTtlConfig.UpdateType.OnCreateAndWrite` - only on creation and write
access,
+ - `StateTtlConfig.UpdateType.OnReadAndWrite` - also on read access.
+
+The state visibility configures whether the expired value is returned on read
access
+if it is not cleaned up yet (default `NeverReturnExpired`):
+
+ - `StateTtlConfig.StateVisibility.NeverReturnExpired` - expired value is
never returned,
+ - `StateTtlConfig.StateVisibility.ReturnExpiredIfNotCleanedUp` - returned if
still available.
+
+ In case of `NeverReturnExpired`, the expired state behaves as if it does not
exist anymore,
+ even if it has yet to be removed. The option can be useful for the use cases
+ where data has to become unavailable for read access strictly after TTL,
+ e.g. application working with privacy sensitive data.
+
+Another option `ReturnExpiredIfNotCleanedUp` allows to return the expired
state before its cleanup.
+
+**Notes:**
+
+- The state backends store the timestamp of last modification along with the
user value,
+which means that enabling this feature increases consumption of state storage.
+Heap state backend stores an additional Java object with a reference to the
user state object
+and a primitive long value in memory. The RocksDB state backend adds 8 bytes
per stored value, list entry or map entry.
+
+- Only TTLs in reference to *processing time* are currently supported.
+
+- Trying to restore state, which was previously configured without TTL, using
TTL enabled descriptor or vice versa
+will lead to compatibility failure and `StateMigrationException`.
+
+- The TTL configuration is not part of check- or savepoints
+but rather a way how Flink treats it in the currently running job.
+
+ Cleanup of expired state
+
+Currently expired values are always removed when they are read out explicitly,
+e.g. by calling `ValueState.value()`.
+
+Attention! This means that by default
if expired
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[ https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574837#comment-16574837 ] ASF GitHub Bot commented on FLINK-9637: --- twalthr commented on issue #6379: [FLINK-9637] Add public user documentation for state TTL feature URL: https://github.com/apache/flink/pull/6379#issuecomment-411756912 Thank you @azagrebin. I will go through the text a last time and merge this... This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add public user documentation for TTL feature > - > > Key: FLINK-9637 > URL: https://issues.apache.org/jira/browse/FLINK-9637 > Project: Flink > Issue Type: Sub-task > Components: State Backends, Checkpointing >Affects Versions: 1.6.0 >Reporter: Andrey Zagrebin >Assignee: Andrey Zagrebin >Priority: Major > Labels: pull-request-available > Fix For: 1.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[ https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571691#comment-16571691 ] ASF GitHub Bot commented on FLINK-9637: --- azagrebin commented on issue #6379: [FLINK-9637] Add public user documentation for state TTL feature URL: https://github.com/apache/flink/pull/6379#issuecomment-411065564 Thank you for the review @twalthr, I updated the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add public user documentation for TTL feature > - > > Key: FLINK-9637 > URL: https://issues.apache.org/jira/browse/FLINK-9637 > Project: Flink > Issue Type: Sub-task > Components: State Backends, Checkpointing >Affects Versions: 1.6.0 >Reporter: Andrey Zagrebin >Assignee: Andrey Zagrebin >Priority: Major > Labels: pull-request-available > Fix For: 1.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[
https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571277#comment-16571277
]
ASF GitHub Bot commented on FLINK-9637:
---
twalthr commented on a change in pull request #6379: [FLINK-9637] Add public
user documentation for state TTL feature
URL: https://github.com/apache/flink/pull/6379#discussion_r208143071
##
File path: docs/dev/stream/state/state.md
##
@@ -266,6 +266,101 @@ a `ValueState`. Once the count reaches 2 it will emit
the average and clear the
we start over from `0`. Note that this would keep a different state value for
each different input
key if we had tuples with different values in the first field.
+### State time-to-live (TTL)
+
+A time-to-live (TTL) can be assigned to the keyed state value.
+In this case it will expire after the configured TTL
+and its stored value will be cleaned up on the best effort basis which is
discussed in details later.
+
+The state collection types support per-entry TTLs: list elements and map
entries expire independently.
+
+To use state TTL you must first build a `StateTtlConfiguration` object:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build();
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build()
+{% endhighlight %}
+
+
+
+It has several options to consider.
+The first parameter of `newBuilder` method is mandatory, it is the
time-to-live value.
+
+The update type configures when the state TTL is refreshed (default
`OnCreateAndWrite`):
+
+ - `StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite` - only on creation
and write access,
+ - `StateTtlConfiguration.TtlUpdateType.OnReadAndWrite` - also on read access.
+
+The state visibility configures whether the expired value is returned on read
access
+if it is not cleaned up yet (default `NeverReturnExpired`):
+
+ - `StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired` - expired
value is never returned,
+ - `StateTtlConfiguration.TtlStateVisibility.ReturnExpiredIfNotCleanedUp` -
returned if still available.
+
+ In case of `NeverReturnExpired`, the expired state behaves as if it does not
exist any more,
+ even if it has yet to be removed. The option can be useful for the use cases
+ where data has to become unavailable for read access strictly after TTL,
+ e.g. application working with privacy sensitive data.
+
+Another option `ReturnExpiredIfNotCleanedUp` allows to return the expired
state before its cleanup.
+
+TTL functionality can be enabled in the descriptor of any type of state:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig =
StateTtlConfiguration.newBuilder(Time.seconds(1)).build();
+ValueStateDescriptor stateDescriptor = new
ValueStateDescriptor<>("text state", String.class);
+stateDescriptor.enableTimeToLive(ttlConfig);
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration.newBuilder(Time.seconds(1)).build()
+val stateDescriptor = new ValueStateDescriptor[String]("text state",
classOf[String])
+stateDescriptor.enableTimeToLive(ttlConfig)
+{% endhighlight %}
+
+
+
+**Notes:**
+
+- The state backends store the timestamp of last modification along with the
user value,
+which means that enabling this feature increases consumption of state storage.
+Heap state backend stores an additional java object with a reference to the
user state object
Review comment:
`Java`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add public user documentation for TTL feature
> -
>
> Key: FLINK-9637
> URL: https://issues.apache.org/jira/browse/FLINK-9637
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
>Affects Versions: 1.6.0
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[ https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571272#comment-16571272 ] ASF GitHub Bot commented on FLINK-9637: --- twalthr commented on a change in pull request #6379: [FLINK-9637] Add public user documentation for state TTL feature URL: https://github.com/apache/flink/pull/6379#discussion_r208138767 ## File path: docs/dev/stream/state/state.md ## @@ -266,6 +266,101 @@ a `ValueState`. Once the count reaches 2 it will emit the average and clear the we start over from `0`. Note that this would keep a different state value for each different input key if we had tuples with different values in the first field. +### State time-to-live (TTL) + +A time-to-live (TTL) can be assigned to the keyed state value. Review comment: `to each keyed state value` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add public user documentation for TTL feature > - > > Key: FLINK-9637 > URL: https://issues.apache.org/jira/browse/FLINK-9637 > Project: Flink > Issue Type: Sub-task > Components: State Backends, Checkpointing >Affects Versions: 1.6.0 >Reporter: Andrey Zagrebin >Assignee: Andrey Zagrebin >Priority: Major > Labels: pull-request-available > Fix For: 1.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[
https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571281#comment-16571281
]
ASF GitHub Bot commented on FLINK-9637:
---
twalthr commented on a change in pull request #6379: [FLINK-9637] Add public
user documentation for state TTL feature
URL: https://github.com/apache/flink/pull/6379#discussion_r208143373
##
File path: docs/dev/stream/state/state.md
##
@@ -266,6 +266,101 @@ a `ValueState`. Once the count reaches 2 it will emit
the average and clear the
we start over from `0`. Note that this would keep a different state value for
each different input
key if we had tuples with different values in the first field.
+### State time-to-live (TTL)
+
+A time-to-live (TTL) can be assigned to the keyed state value.
+In this case it will expire after the configured TTL
+and its stored value will be cleaned up on the best effort basis which is
discussed in details later.
+
+The state collection types support per-entry TTLs: list elements and map
entries expire independently.
+
+To use state TTL you must first build a `StateTtlConfiguration` object:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build();
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build()
+{% endhighlight %}
+
+
+
+It has several options to consider.
+The first parameter of `newBuilder` method is mandatory, it is the
time-to-live value.
+
+The update type configures when the state TTL is refreshed (default
`OnCreateAndWrite`):
+
+ - `StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite` - only on creation
and write access,
+ - `StateTtlConfiguration.TtlUpdateType.OnReadAndWrite` - also on read access.
+
+The state visibility configures whether the expired value is returned on read
access
+if it is not cleaned up yet (default `NeverReturnExpired`):
+
+ - `StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired` - expired
value is never returned,
+ - `StateTtlConfiguration.TtlStateVisibility.ReturnExpiredIfNotCleanedUp` -
returned if still available.
+
+ In case of `NeverReturnExpired`, the expired state behaves as if it does not
exist any more,
+ even if it has yet to be removed. The option can be useful for the use cases
+ where data has to become unavailable for read access strictly after TTL,
+ e.g. application working with privacy sensitive data.
+
+Another option `ReturnExpiredIfNotCleanedUp` allows to return the expired
state before its cleanup.
+
+TTL functionality can be enabled in the descriptor of any type of state:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig =
StateTtlConfiguration.newBuilder(Time.seconds(1)).build();
+ValueStateDescriptor stateDescriptor = new
ValueStateDescriptor<>("text state", String.class);
+stateDescriptor.enableTimeToLive(ttlConfig);
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration.newBuilder(Time.seconds(1)).build()
+val stateDescriptor = new ValueStateDescriptor[String]("text state",
classOf[String])
+stateDescriptor.enableTimeToLive(ttlConfig)
+{% endhighlight %}
+
+
+
+**Notes:**
+
+- The state backends store the timestamp of last modification along with the
user value,
+which means that enabling this feature increases consumption of state storage.
+Heap state backend stores an additional java object with a reference to the
user state object
+and a primitive long in memory. RocksDB state backend adds 8 bytes per stored
value, list entry or map entry.
Review comment:
`The RocksDB state backend`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add public user documentation for TTL feature
> -
>
> Key: FLINK-9637
> URL: https://issues.apache.org/jira/browse/FLINK-9637
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
>Affects Versions: 1.6.0
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[
https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571279#comment-16571279
]
ASF GitHub Bot commented on FLINK-9637:
---
twalthr commented on a change in pull request #6379: [FLINK-9637] Add public
user documentation for state TTL feature
URL: https://github.com/apache/flink/pull/6379#discussion_r208143176
##
File path: docs/dev/stream/state/state.md
##
@@ -266,6 +266,101 @@ a `ValueState`. Once the count reaches 2 it will emit
the average and clear the
we start over from `0`. Note that this would keep a different state value for
each different input
key if we had tuples with different values in the first field.
+### State time-to-live (TTL)
+
+A time-to-live (TTL) can be assigned to the keyed state value.
+In this case it will expire after the configured TTL
+and its stored value will be cleaned up on the best effort basis which is
discussed in details later.
+
+The state collection types support per-entry TTLs: list elements and map
entries expire independently.
+
+To use state TTL you must first build a `StateTtlConfiguration` object:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build();
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build()
+{% endhighlight %}
+
+
+
+It has several options to consider.
+The first parameter of `newBuilder` method is mandatory, it is the
time-to-live value.
+
+The update type configures when the state TTL is refreshed (default
`OnCreateAndWrite`):
+
+ - `StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite` - only on creation
and write access,
+ - `StateTtlConfiguration.TtlUpdateType.OnReadAndWrite` - also on read access.
+
+The state visibility configures whether the expired value is returned on read
access
+if it is not cleaned up yet (default `NeverReturnExpired`):
+
+ - `StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired` - expired
value is never returned,
+ - `StateTtlConfiguration.TtlStateVisibility.ReturnExpiredIfNotCleanedUp` -
returned if still available.
+
+ In case of `NeverReturnExpired`, the expired state behaves as if it does not
exist any more,
+ even if it has yet to be removed. The option can be useful for the use cases
+ where data has to become unavailable for read access strictly after TTL,
+ e.g. application working with privacy sensitive data.
+
+Another option `ReturnExpiredIfNotCleanedUp` allows to return the expired
state before its cleanup.
+
+TTL functionality can be enabled in the descriptor of any type of state:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig =
StateTtlConfiguration.newBuilder(Time.seconds(1)).build();
+ValueStateDescriptor stateDescriptor = new
ValueStateDescriptor<>("text state", String.class);
+stateDescriptor.enableTimeToLive(ttlConfig);
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration.newBuilder(Time.seconds(1)).build()
+val stateDescriptor = new ValueStateDescriptor[String]("text state",
classOf[String])
+stateDescriptor.enableTimeToLive(ttlConfig)
+{% endhighlight %}
+
+
+
+**Notes:**
+
+- The state backends store the timestamp of last modification along with the
user value,
+which means that enabling this feature increases consumption of state storage.
+Heap state backend stores an additional java object with a reference to the
user state object
+and a primitive long in memory. RocksDB state backend adds 8 bytes per stored
value, list entry or map entry.
Review comment:
`long value`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add public user documentation for TTL feature
> -
>
> Key: FLINK-9637
> URL: https://issues.apache.org/jira/browse/FLINK-9637
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
>Affects Versions: 1.6.0
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[
https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571275#comment-16571275
]
ASF GitHub Bot commented on FLINK-9637:
---
twalthr commented on a change in pull request #6379: [FLINK-9637] Add public
user documentation for state TTL feature
URL: https://github.com/apache/flink/pull/6379#discussion_r208140806
##
File path: docs/dev/stream/state/state.md
##
@@ -266,6 +266,101 @@ a `ValueState`. Once the count reaches 2 it will emit
the average and clear the
we start over from `0`. Note that this would keep a different state value for
each different input
key if we had tuples with different values in the first field.
+### State time-to-live (TTL)
+
+A time-to-live (TTL) can be assigned to the keyed state value.
+In this case it will expire after the configured TTL
+and its stored value will be cleaned up on the best effort basis which is
discussed in details later.
+
+The state collection types support per-entry TTLs: list elements and map
entries expire independently.
+
+To use state TTL you must first build a `StateTtlConfiguration` object:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build();
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build()
+{% endhighlight %}
+
+
+
+It has several options to consider.
+The first parameter of `newBuilder` method is mandatory, it is the
time-to-live value.
+
+The update type configures when the state TTL is refreshed (default
`OnCreateAndWrite`):
+
+ - `StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite` - only on creation
and write access,
+ - `StateTtlConfiguration.TtlUpdateType.OnReadAndWrite` - also on read access.
+
+The state visibility configures whether the expired value is returned on read
access
+if it is not cleaned up yet (default `NeverReturnExpired`):
+
+ - `StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired` - expired
value is never returned,
+ - `StateTtlConfiguration.TtlStateVisibility.ReturnExpiredIfNotCleanedUp` -
returned if still available.
Review comment:
IMHO we could have shorted the names a little bit and moved
`TtlStateVisibility` to a separate class. But now it is too late.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add public user documentation for TTL feature
> -
>
> Key: FLINK-9637
> URL: https://issues.apache.org/jira/browse/FLINK-9637
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
>Affects Versions: 1.6.0
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[ https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571276#comment-16571276 ] ASF GitHub Bot commented on FLINK-9637: --- twalthr commented on a change in pull request #6379: [FLINK-9637] Add public user documentation for state TTL feature URL: https://github.com/apache/flink/pull/6379#discussion_r208142378 ## File path: docs/dev/stream/state/state.md ## @@ -266,6 +266,101 @@ a `ValueState`. Once the count reaches 2 it will emit the average and clear the we start over from `0`. Note that this would keep a different state value for each different input key if we had tuples with different values in the first field. +### State time-to-live (TTL) + +A time-to-live (TTL) can be assigned to the keyed state value. +In this case it will expire after the configured TTL +and its stored value will be cleaned up on the best effort basis which is discussed in details later. + +The state collection types support per-entry TTLs: list elements and map entries expire independently. + +To use state TTL you must first build a `StateTtlConfiguration` object: + + Review comment: Show in the first example how this feature is used. Not only the configuration without context. Maybe with a example state for which the config is set. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add public user documentation for TTL feature > - > > Key: FLINK-9637 > URL: https://issues.apache.org/jira/browse/FLINK-9637 > Project: Flink > Issue Type: Sub-task > Components: State Backends, Checkpointing >Affects Versions: 1.6.0 >Reporter: Andrey Zagrebin >Assignee: Andrey Zagrebin >Priority: Major > Labels: pull-request-available > Fix For: 1.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[
https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571271#comment-16571271
]
ASF GitHub Bot commented on FLINK-9637:
---
twalthr commented on a change in pull request #6379: [FLINK-9637] Add public
user documentation for state TTL feature
URL: https://github.com/apache/flink/pull/6379#discussion_r208139216
##
File path: docs/dev/stream/state/state.md
##
@@ -266,6 +266,101 @@ a `ValueState`. Once the count reaches 2 it will emit
the average and clear the
we start over from `0`. Note that this would keep a different state value for
each different input
key if we had tuples with different values in the first field.
+### State time-to-live (TTL)
+
+A time-to-live (TTL) can be assigned to the keyed state value.
+In this case it will expire after the configured TTL
+and its stored value will be cleaned up on the best effort basis which is
discussed in details later.
+
+The state collection types support per-entry TTLs: list elements and map
entries expire independently.
+
+To use state TTL you must first build a `StateTtlConfiguration` object:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig = StateTtlConfiguration
Review comment:
Include imports to find the newly introduced class. Also to clarify which
Flink `Time` class we are using.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add public user documentation for TTL feature
> -
>
> Key: FLINK-9637
> URL: https://issues.apache.org/jira/browse/FLINK-9637
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
>Affects Versions: 1.6.0
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[
https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571282#comment-16571282
]
ASF GitHub Bot commented on FLINK-9637:
---
twalthr commented on a change in pull request #6379: [FLINK-9637] Add public
user documentation for state TTL feature
URL: https://github.com/apache/flink/pull/6379#discussion_r208144366
##
File path: docs/dev/stream/state/state.md
##
@@ -266,6 +266,101 @@ a `ValueState`. Once the count reaches 2 it will emit
the average and clear the
we start over from `0`. Note that this would keep a different state value for
each different input
key if we had tuples with different values in the first field.
+### State time-to-live (TTL)
+
+A time-to-live (TTL) can be assigned to the keyed state value.
+In this case it will expire after the configured TTL
+and its stored value will be cleaned up on the best effort basis which is
discussed in details later.
+
+The state collection types support per-entry TTLs: list elements and map
entries expire independently.
+
+To use state TTL you must first build a `StateTtlConfiguration` object:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build();
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build()
+{% endhighlight %}
+
+
+
+It has several options to consider.
+The first parameter of `newBuilder` method is mandatory, it is the
time-to-live value.
+
+The update type configures when the state TTL is refreshed (default
`OnCreateAndWrite`):
+
+ - `StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite` - only on creation
and write access,
+ - `StateTtlConfiguration.TtlUpdateType.OnReadAndWrite` - also on read access.
+
+The state visibility configures whether the expired value is returned on read
access
+if it is not cleaned up yet (default `NeverReturnExpired`):
+
+ - `StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired` - expired
value is never returned,
+ - `StateTtlConfiguration.TtlStateVisibility.ReturnExpiredIfNotCleanedUp` -
returned if still available.
+
+ In case of `NeverReturnExpired`, the expired state behaves as if it does not
exist any more,
+ even if it has yet to be removed. The option can be useful for the use cases
+ where data has to become unavailable for read access strictly after TTL,
+ e.g. application working with privacy sensitive data.
+
+Another option `ReturnExpiredIfNotCleanedUp` allows to return the expired
state before its cleanup.
+
+TTL functionality can be enabled in the descriptor of any type of state:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig =
StateTtlConfiguration.newBuilder(Time.seconds(1)).build();
+ValueStateDescriptor stateDescriptor = new
ValueStateDescriptor<>("text state", String.class);
+stateDescriptor.enableTimeToLive(ttlConfig);
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration.newBuilder(Time.seconds(1)).build()
+val stateDescriptor = new ValueStateDescriptor[String]("text state",
classOf[String])
+stateDescriptor.enableTimeToLive(ttlConfig)
+{% endhighlight %}
+
+
+
+**Notes:**
+
+- The state backends store the timestamp of last modification along with the
user value,
+which means that enabling this feature increases consumption of state storage.
+Heap state backend stores an additional java object with a reference to the
user state object
+and a primitive long in memory. RocksDB state backend adds 8 bytes per stored
value, list entry or map entry.
+
+- Currently expired values are only removed when they are read out explicitly,
+e.g. by calling ValueState.value().
+Note that this means that under the current implementation if expired state is
not read,
+it won't be removed, possibly leading to ever growing state.
+This should change in future releases.
+Additional strategies should be added that clean up expired state
automatically in the background.
+
+- Only TTLs in reference to *processing time* are currently supported.
+
+- Trying to restore state, which was previously configured without TTL, using
TTL enabled descriptor or vice versa
+will lead to compatibility failure and `StateMigrationException`.
+
+- The TTL configuration is not part of check- or savepoint
Review comment:
`savepoints`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[
https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571274#comment-16571274
]
ASF GitHub Bot commented on FLINK-9637:
---
twalthr commented on a change in pull request #6379: [FLINK-9637] Add public
user documentation for state TTL feature
URL: https://github.com/apache/flink/pull/6379#discussion_r208144581
##
File path: docs/dev/table/index.md
##
@@ -79,4 +79,4 @@ Where to go next?
* [Table Sources & Sinks]({{ site.baseurl }}/dev/table/sourceSinks.html):
Reading tables from and emitting tables to external storage systems.
* [User-Defined Functions]({{ site.baseurl }}/dev/table/udfs.html): Definition
and usage of user-defined functions.
-{% top %}
\ No newline at end of file
+{% top %}
Review comment:
Why did you remove that?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add public user documentation for TTL feature
> -
>
> Key: FLINK-9637
> URL: https://issues.apache.org/jira/browse/FLINK-9637
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
>Affects Versions: 1.6.0
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[
https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571273#comment-16571273
]
ASF GitHub Bot commented on FLINK-9637:
---
twalthr commented on a change in pull request #6379: [FLINK-9637] Add public
user documentation for state TTL feature
URL: https://github.com/apache/flink/pull/6379#discussion_r208139795
##
File path: docs/dev/stream/state/state.md
##
@@ -266,6 +266,101 @@ a `ValueState`. Once the count reaches 2 it will emit
the average and clear the
we start over from `0`. Note that this would keep a different state value for
each different input
key if we had tuples with different values in the first field.
+### State time-to-live (TTL)
+
+A time-to-live (TTL) can be assigned to the keyed state value.
+In this case it will expire after the configured TTL
+and its stored value will be cleaned up on the best effort basis which is
discussed in details later.
+
+The state collection types support per-entry TTLs: list elements and map
entries expire independently.
+
+To use state TTL you must first build a `StateTtlConfiguration` object:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build();
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build()
+{% endhighlight %}
+
+
+
+It has several options to consider.
Review comment:
What is "it"?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add public user documentation for TTL feature
> -
>
> Key: FLINK-9637
> URL: https://issues.apache.org/jira/browse/FLINK-9637
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
>Affects Versions: 1.6.0
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[
https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571283#comment-16571283
]
ASF GitHub Bot commented on FLINK-9637:
---
twalthr commented on a change in pull request #6379: [FLINK-9637] Add public
user documentation for state TTL feature
URL: https://github.com/apache/flink/pull/6379#discussion_r208142591
##
File path: docs/dev/stream/state/state.md
##
@@ -266,6 +266,101 @@ a `ValueState`. Once the count reaches 2 it will emit
the average and clear the
we start over from `0`. Note that this would keep a different state value for
each different input
key if we had tuples with different values in the first field.
+### State time-to-live (TTL)
+
+A time-to-live (TTL) can be assigned to the keyed state value.
+In this case it will expire after the configured TTL
+and its stored value will be cleaned up on the best effort basis which is
discussed in details later.
+
+The state collection types support per-entry TTLs: list elements and map
entries expire independently.
+
+To use state TTL you must first build a `StateTtlConfiguration` object:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build();
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build()
+{% endhighlight %}
+
+
+
+It has several options to consider.
+The first parameter of `newBuilder` method is mandatory, it is the
time-to-live value.
+
+The update type configures when the state TTL is refreshed (default
`OnCreateAndWrite`):
+
+ - `StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite` - only on creation
and write access,
+ - `StateTtlConfiguration.TtlUpdateType.OnReadAndWrite` - also on read access.
+
+The state visibility configures whether the expired value is returned on read
access
+if it is not cleaned up yet (default `NeverReturnExpired`):
+
+ - `StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired` - expired
value is never returned,
+ - `StateTtlConfiguration.TtlStateVisibility.ReturnExpiredIfNotCleanedUp` -
returned if still available.
+
+ In case of `NeverReturnExpired`, the expired state behaves as if it does not
exist any more,
+ even if it has yet to be removed. The option can be useful for the use cases
+ where data has to become unavailable for read access strictly after TTL,
+ e.g. application working with privacy sensitive data.
+
+Another option `ReturnExpiredIfNotCleanedUp` allows to return the expired
state before its cleanup.
+
+TTL functionality can be enabled in the descriptor of any type of state:
Review comment:
We could move this example to the beginning?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add public user documentation for TTL feature
> -
>
> Key: FLINK-9637
> URL: https://issues.apache.org/jira/browse/FLINK-9637
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
>Affects Versions: 1.6.0
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[
https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571280#comment-16571280
]
ASF GitHub Bot commented on FLINK-9637:
---
twalthr commented on a change in pull request #6379: [FLINK-9637] Add public
user documentation for state TTL feature
URL: https://github.com/apache/flink/pull/6379#discussion_r208143812
##
File path: docs/dev/stream/state/state.md
##
@@ -266,6 +266,101 @@ a `ValueState`. Once the count reaches 2 it will emit
the average and clear the
we start over from `0`. Note that this would keep a different state value for
each different input
key if we had tuples with different values in the first field.
+### State time-to-live (TTL)
+
+A time-to-live (TTL) can be assigned to the keyed state value.
+In this case it will expire after the configured TTL
+and its stored value will be cleaned up on the best effort basis which is
discussed in details later.
+
+The state collection types support per-entry TTLs: list elements and map
entries expire independently.
+
+To use state TTL you must first build a `StateTtlConfiguration` object:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build();
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build()
+{% endhighlight %}
+
+
+
+It has several options to consider.
+The first parameter of `newBuilder` method is mandatory, it is the
time-to-live value.
+
+The update type configures when the state TTL is refreshed (default
`OnCreateAndWrite`):
+
+ - `StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite` - only on creation
and write access,
+ - `StateTtlConfiguration.TtlUpdateType.OnReadAndWrite` - also on read access.
+
+The state visibility configures whether the expired value is returned on read
access
+if it is not cleaned up yet (default `NeverReturnExpired`):
+
+ - `StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired` - expired
value is never returned,
+ - `StateTtlConfiguration.TtlStateVisibility.ReturnExpiredIfNotCleanedUp` -
returned if still available.
+
+ In case of `NeverReturnExpired`, the expired state behaves as if it does not
exist any more,
+ even if it has yet to be removed. The option can be useful for the use cases
+ where data has to become unavailable for read access strictly after TTL,
+ e.g. application working with privacy sensitive data.
+
+Another option `ReturnExpiredIfNotCleanedUp` allows to return the expired
state before its cleanup.
+
+TTL functionality can be enabled in the descriptor of any type of state:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig =
StateTtlConfiguration.newBuilder(Time.seconds(1)).build();
+ValueStateDescriptor stateDescriptor = new
ValueStateDescriptor<>("text state", String.class);
+stateDescriptor.enableTimeToLive(ttlConfig);
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration.newBuilder(Time.seconds(1)).build()
+val stateDescriptor = new ValueStateDescriptor[String]("text state",
classOf[String])
+stateDescriptor.enableTimeToLive(ttlConfig)
+{% endhighlight %}
+
+
+
+**Notes:**
+
+- The state backends store the timestamp of last modification along with the
user value,
+which means that enabling this feature increases consumption of state storage.
+Heap state backend stores an additional java object with a reference to the
user state object
+and a primitive long in memory. RocksDB state backend adds 8 bytes per stored
value, list entry or map entry.
+
+- Currently expired values are only removed when they are read out explicitly,
Review comment:
We should mentione this more explicit at the beginning. This is too hidden
in my opinion. Use `Attention`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add public user documentation for TTL feature
> -
>
> Key: FLINK-9637
> URL: https://issues.apache.org/jira/browse/FLINK-9637
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
>Affects Versions: 1.6.0
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[
https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571278#comment-16571278
]
ASF GitHub Bot commented on FLINK-9637:
---
twalthr commented on a change in pull request #6379: [FLINK-9637] Add public
user documentation for state TTL feature
URL: https://github.com/apache/flink/pull/6379#discussion_r208141488
##
File path: docs/dev/stream/state/state.md
##
@@ -266,6 +266,101 @@ a `ValueState`. Once the count reaches 2 it will emit
the average and clear the
we start over from `0`. Note that this would keep a different state value for
each different input
key if we had tuples with different values in the first field.
+### State time-to-live (TTL)
+
+A time-to-live (TTL) can be assigned to the keyed state value.
+In this case it will expire after the configured TTL
+and its stored value will be cleaned up on the best effort basis which is
discussed in details later.
+
+The state collection types support per-entry TTLs: list elements and map
entries expire independently.
+
+To use state TTL you must first build a `StateTtlConfiguration` object:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build();
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build()
+{% endhighlight %}
+
+
+
+It has several options to consider.
+The first parameter of `newBuilder` method is mandatory, it is the
time-to-live value.
+
+The update type configures when the state TTL is refreshed (default
`OnCreateAndWrite`):
+
+ - `StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite` - only on creation
and write access,
+ - `StateTtlConfiguration.TtlUpdateType.OnReadAndWrite` - also on read access.
+
+The state visibility configures whether the expired value is returned on read
access
+if it is not cleaned up yet (default `NeverReturnExpired`):
+
+ - `StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired` - expired
value is never returned,
+ - `StateTtlConfiguration.TtlStateVisibility.ReturnExpiredIfNotCleanedUp` -
returned if still available.
+
+ In case of `NeverReturnExpired`, the expired state behaves as if it does not
exist any more,
Review comment:
`anymore`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add public user documentation for TTL feature
> -
>
> Key: FLINK-9637
> URL: https://issues.apache.org/jira/browse/FLINK-9637
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
>Affects Versions: 1.6.0
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[ https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568178#comment-16568178 ] ASF GitHub Bot commented on FLINK-9637: --- azagrebin commented on issue #6379: [FLINK-9637] Add public user documentation for state TTL feature URL: https://github.com/apache/flink/pull/6379#issuecomment-410246921 Thanks for review guys! I updated the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add public user documentation for TTL feature > - > > Key: FLINK-9637 > URL: https://issues.apache.org/jira/browse/FLINK-9637 > Project: Flink > Issue Type: Sub-task > Components: State Backends, Checkpointing >Affects Versions: 1.6.0 >Reporter: Andrey Zagrebin >Assignee: Andrey Zagrebin >Priority: Major > Labels: pull-request-available > Fix For: 1.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[
https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568177#comment-16568177
]
ASF GitHub Bot commented on FLINK-9637:
---
azagrebin commented on a change in pull request #6379: [FLINK-9637] Add public
user documentation for state TTL feature
URL: https://github.com/apache/flink/pull/6379#discussion_r207537238
##
File path: docs/dev/stream/state/state.md
##
@@ -266,6 +266,92 @@ a `ValueState`. Once the count reaches 2 it will emit the
average and clear the
we start over from `0`. Note that this would keep a different state value for
each different input
key if we had tuples with different values in the first field.
+### State time-to-live (TTL)
+
+A time-to-live (TTL) can be assigned to the keyed state value.
+In this case it will expire after the configured TTL
+and its stored value will be cleaned up based on the best effort.
+Depending on configuration, the expired state can become unavailable for read
access
+even if it is not cleaned up yet. In this case it behaves as if it does not
exist any more.
+
+The collection types of state support TTL on entry level:
+separate list elements and map entries expire independently.
+
+The behaviour of state with TTL firstly should be configured by building
`StateTtlConfiguration`:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
Review comment:
Agree, I added the renaming commit in PR #6460 for FLINK-9938.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add public user documentation for TTL feature
> -
>
> Key: FLINK-9637
> URL: https://issues.apache.org/jira/browse/FLINK-9637
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
>Affects Versions: 1.6.0
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[
https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568175#comment-16568175
]
ASF GitHub Bot commented on FLINK-9637:
---
azagrebin commented on a change in pull request #6379: [FLINK-9637] Add public
user documentation for state TTL feature
URL: https://github.com/apache/flink/pull/6379#discussion_r207536863
##
File path: docs/dev/stream/state/state.md
##
@@ -266,6 +266,92 @@ a `ValueState`. Once the count reaches 2 it will emit the
average and clear the
we start over from `0`. Note that this would keep a different state value for
each different input
key if we had tuples with different values in the first field.
+### State time-to-live (TTL)
+
+A time-to-live (TTL) can be assigned to the keyed state value.
+In this case it will expire after the configured TTL
+and its stored value will be cleaned up based on the best effort.
+Depending on configuration, the expired state can become unavailable for read
access
+even if it is not cleaned up yet. In this case it behaves as if it does not
exist any more.
+
+The collection types of state support TTL on entry level:
+separate list elements and map entries expire independently.
+
+The behaviour of state with TTL firstly should be configured by building
`StateTtlConfiguration`:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build();
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build()
+{% endhighlight %}
+
+
+
+It has several options to consider.
+The first parameter of `newBuilder` method is mandatory, it is a value of
time-to-live itself.
+
+The update type configures when the time-to-live of state value is prolonged
(default `OnCreateAndWrite`):
+
+ - `StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite` - only on creation
and write access,
+ - `StateTtlConfiguration.TtlUpdateType.OnReadAndWrite` - also on read access.
+
+The state visibility configures whether the expired value is returned on read
access
+if it is not cleaned up yet (default `NeverReturnExpired`):
+
+ - `StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired` - expired
value is never returned,
+ - `StateTtlConfiguration.TtlStateVisibility.ReturnExpiredIfNotCleanedUp` -
returned if still available.
+
+The TTL can be enabled in descriptor for any type of state:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig =
StateTtlConfiguration.newBuilder(Time.seconds(1)).build();
+ValueStateDescriptor stateDescriptor = new
ValueStateDescriptor<>("text state", String.class);
+stateDescriptor.enableTimeToLive(ttlConfig);
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration.newBuilder(Time.seconds(1)).build()
+val stateDescriptor = new ValueStateDescriptor[String]("text state",
classOf[String])
+stateDescriptor.enableTimeToLive(ttlConfig)
+{% endhighlight %}
+
+
+
+**Notes:**
+
+- The state backends store the timestamp of last modification along with the
user value,
+which means that enabling this feature increases consumption of state storage.
+
+- As of current implementation the state storage is cleaned up of expired
value
+only on its explicit read access per key, e.g. calling `ValueState.value()`.
+This might change in future releases, e.g. additional strategies might be
added in background to speed up cleanup.
Review comment:
I think we still want to improve this, because FLINK-9938 is limited in a
way that it cleans up only the snapshot data but not the local one used by
running pipeline atm.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add public user documentation for TTL feature
> -
>
> Key: FLINK-9637
> URL: https://issues.apache.org/jira/browse/FLINK-9637
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
>Affects Versions: 1.6.0
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[
https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563163#comment-16563163
]
ASF GitHub Bot commented on FLINK-9637:
---
bowenli86 commented on a change in pull request #6379: [FLINK-9637] Add public
user documentation for state TTL feature
URL: https://github.com/apache/flink/pull/6379#discussion_r206399972
##
File path: docs/dev/stream/state/state.md
##
@@ -266,6 +266,92 @@ a `ValueState`. Once the count reaches 2 it will emit the
average and clear the
we start over from `0`. Note that this would keep a different state value for
each different input
key if we had tuples with different values in the first field.
+### State time-to-live (TTL)
+
+A time-to-live (TTL) can be assigned to the keyed state value.
+In this case it will expire after the configured TTL
+and its stored value will be cleaned up based on the best effort.
+Depending on configuration, the expired state can become unavailable for read
access
+even if it is not cleaned up yet. In this case it behaves as if it does not
exist any more.
+
+The collection types of state support TTL on entry level:
+separate list elements and map entries expire independently.
+
+The behaviour of state with TTL firstly should be configured by building
`StateTtlConfiguration`:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
Review comment:
My gut feeling is that the class names are quite verbose currently - like
duplicated 'Ttl' in the namings. If it's not too late for release 1.6.0, would
be nice to shorten it to something like
`StateTtlConfig.UpdateType.OnCreateAndWrite`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add public user documentation for TTL feature
> -
>
> Key: FLINK-9637
> URL: https://issues.apache.org/jira/browse/FLINK-9637
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
>Affects Versions: 1.6.0
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[
https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561605#comment-16561605
]
ASF GitHub Bot commented on FLINK-9637:
---
StefanRRichter commented on a change in pull request #6379: [FLINK-9637] Add
public user documentation for state TTL feature
URL: https://github.com/apache/flink/pull/6379#discussion_r206044480
##
File path: docs/dev/stream/state/state.md
##
@@ -266,6 +266,92 @@ a `ValueState`. Once the count reaches 2 it will emit the
average and clear the
we start over from `0`. Note that this would keep a different state value for
each different input
key if we had tuples with different values in the first field.
+### State time-to-live (TTL)
+
+A time-to-live (TTL) can be assigned to the keyed state value.
+In this case it will expire after the configured TTL
+and its stored value will be cleaned up based on the best effort.
+Depending on configuration, the expired state can become unavailable for read
access
+even if it is not cleaned up yet. In this case it behaves as if it does not
exist any more.
+
+The collection types of state support TTL on entry level:
+separate list elements and map entries expire independently.
+
+The behaviour of state with TTL firstly should be configured by building
`StateTtlConfiguration`:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build();
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build()
+{% endhighlight %}
+
+
+
+It has several options to consider.
+The first parameter of `newBuilder` method is mandatory, it is a value of
time-to-live itself.
+
+The update type configures when the time-to-live of state value is prolonged
(default `OnCreateAndWrite`):
+
+ - `StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite` - only on creation
and write access,
+ - `StateTtlConfiguration.TtlUpdateType.OnReadAndWrite` - also on read access.
+
+The state visibility configures whether the expired value is returned on read
access
+if it is not cleaned up yet (default `NeverReturnExpired`):
+
+ - `StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired` - expired
value is never returned,
+ - `StateTtlConfiguration.TtlStateVisibility.ReturnExpiredIfNotCleanedUp` -
returned if still available.
+
+The TTL can be enabled in descriptor for any type of state:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig =
StateTtlConfiguration.newBuilder(Time.seconds(1)).build();
+ValueStateDescriptor stateDescriptor = new
ValueStateDescriptor<>("text state", String.class);
+stateDescriptor.enableTimeToLive(ttlConfig);
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration.newBuilder(Time.seconds(1)).build()
+val stateDescriptor = new ValueStateDescriptor[String]("text state",
classOf[String])
+stateDescriptor.enableTimeToLive(ttlConfig)
+{% endhighlight %}
+
+
+
+**Notes:**
+
+- The state backends store the timestamp of last modification along with the
user value,
+which means that enabling this feature increases consumption of state storage.
+
+- As of current implementation the state storage is cleaned up of expired
value
+only on its explicit read access per key, e.g. calling `ValueState.value()`.
+This might change in future releases, e.g. additional strategies might be
added in background to speed up cleanup.
Review comment:
I think if we merge FLINK-9938 before release, the description should be
updated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add public user documentation for TTL feature
> -
>
> Key: FLINK-9637
> URL: https://issues.apache.org/jira/browse/FLINK-9637
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
>Affects Versions: 1.6.0
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[ https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561604#comment-16561604 ] ASF GitHub Bot commented on FLINK-9637: --- StefanRRichter commented on a change in pull request #6379: [FLINK-9637] Add public user documentation for state TTL feature URL: https://github.com/apache/flink/pull/6379#discussion_r206044982 ## File path: docs/dev/stream/state/state.md ## @@ -266,6 +266,92 @@ a `ValueState`. Once the count reaches 2 it will emit the average and clear the we start over from `0`. Note that this would keep a different state value for each different input key if we had tuples with different values in the first field. +### State time-to-live (TTL) + +A time-to-live (TTL) can be assigned to the keyed state value. +In this case it will expire after the configured TTL +and its stored value will be cleaned up based on the best effort. Review comment: I was expecting some additional info about what "best effort" means here, but it came a bit later. Maybe we should say that this is discussed in more detail later. I think it might also make sense to name some use cases that are currently supported (and some that are not/will be in the future) to make things a bit more concrete for users. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add public user documentation for TTL feature > - > > Key: FLINK-9637 > URL: https://issues.apache.org/jira/browse/FLINK-9637 > Project: Flink > Issue Type: Sub-task > Components: State Backends, Checkpointing >Affects Versions: 1.6.0 >Reporter: Andrey Zagrebin >Assignee: Andrey Zagrebin >Priority: Major > Labels: pull-request-available > Fix For: 1.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[
https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561606#comment-16561606
]
ASF GitHub Bot commented on FLINK-9637:
---
StefanRRichter commented on a change in pull request #6379: [FLINK-9637] Add
public user documentation for state TTL feature
URL: https://github.com/apache/flink/pull/6379#discussion_r206045195
##
File path: docs/dev/stream/state/state.md
##
@@ -266,6 +266,92 @@ a `ValueState`. Once the count reaches 2 it will emit the
average and clear the
we start over from `0`. Note that this would keep a different state value for
each different input
key if we had tuples with different values in the first field.
+### State time-to-live (TTL)
+
+A time-to-live (TTL) can be assigned to the keyed state value.
+In this case it will expire after the configured TTL
+and its stored value will be cleaned up based on the best effort.
+Depending on configuration, the expired state can become unavailable for read
access
+even if it is not cleaned up yet. In this case it behaves as if it does not
exist any more.
+
+The collection types of state support TTL on entry level:
+separate list elements and map entries expire independently.
+
+The behaviour of state with TTL firstly should be configured by building
`StateTtlConfiguration`:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build();
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build()
+{% endhighlight %}
+
+
+
+It has several options to consider.
+The first parameter of `newBuilder` method is mandatory, it is a value of
time-to-live itself.
+
+The update type configures when the time-to-live of state value is prolonged
(default `OnCreateAndWrite`):
+
+ - `StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite` - only on creation
and write access,
+ - `StateTtlConfiguration.TtlUpdateType.OnReadAndWrite` - also on read access.
+
+The state visibility configures whether the expired value is returned on read
access
+if it is not cleaned up yet (default `NeverReturnExpired`):
+
+ - `StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired` - expired
value is never returned,
+ - `StateTtlConfiguration.TtlStateVisibility.ReturnExpiredIfNotCleanedUp` -
returned if still available.
+
+The TTL can be enabled in descriptor for any type of state:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig =
StateTtlConfiguration.newBuilder(Time.seconds(1)).build();
+ValueStateDescriptor stateDescriptor = new
ValueStateDescriptor<>("text state", String.class);
+stateDescriptor.enableTimeToLive(ttlConfig);
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration.newBuilder(Time.seconds(1)).build()
+val stateDescriptor = new ValueStateDescriptor[String]("text state",
classOf[String])
+stateDescriptor.enableTimeToLive(ttlConfig)
+{% endhighlight %}
+
+
+
+**Notes:**
+
+- The state backends store the timestamp of last modification along with the
user value,
+which means that enabling this feature increases consumption of state storage.
+
+- As of current implementation the state storage is cleaned up of expired
value
+only on its explicit read access per key, e.g. calling `ValueState.value()`.
+This might change in future releases, e.g. additional strategies might be
added in background to speed up cleanup.
+
+- Only *processing time* scale is currently supported for TTL.
+
+- Trying to restore state, which was previously configured without TTL, using
TTL enabled descriptor or vice versa
+will lead to compatibility failure.
Review comment:
maybe name the concrete exception
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add public user documentation for TTL feature
> -
>
> Key: FLINK-9637
> URL: https://issues.apache.org/jira/browse/FLINK-9637
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
>Affects Versions: 1.6.0
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
> Labels: pull-request-available
>
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[ https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561603#comment-16561603 ] ASF GitHub Bot commented on FLINK-9637: --- StefanRRichter commented on a change in pull request #6379: [FLINK-9637] Add public user documentation for state TTL feature URL: https://github.com/apache/flink/pull/6379#discussion_r206043626 ## File path: docs/dev/stream/state/state.md ## @@ -266,6 +266,92 @@ a `ValueState`. Once the count reaches 2 it will emit the average and clear the we start over from `0`. Note that this would keep a different state value for each different input key if we had tuples with different values in the first field. +### State time-to-live (TTL) + +A time-to-live (TTL) can be assigned to the keyed state value. +In this case it will expire after the configured TTL +and its stored value will be cleaned up based on the best effort. +Depending on configuration, the expired state can become unavailable for read access +even if it is not cleaned up yet. In this case it behaves as if it does not exist any more. Review comment: This is very abstract in isolation, maybe we should discuss this in the context of the different visibility levels to make it more clear that this is what we are talking about here? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add public user documentation for TTL feature > - > > Key: FLINK-9637 > URL: https://issues.apache.org/jira/browse/FLINK-9637 > Project: Flink > Issue Type: Sub-task > Components: State Backends, Checkpointing >Affects Versions: 1.6.0 >Reporter: Andrey Zagrebin >Assignee: Andrey Zagrebin >Priority: Major > Labels: pull-request-available > Fix For: 1.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[
https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561602#comment-16561602
]
ASF GitHub Bot commented on FLINK-9637:
---
StefanRRichter commented on a change in pull request #6379: [FLINK-9637] Add
public user documentation for state TTL feature
URL: https://github.com/apache/flink/pull/6379#discussion_r206042549
##
File path: docs/dev/stream/state/state.md
##
@@ -266,6 +266,92 @@ a `ValueState`. Once the count reaches 2 it will emit the
average and clear the
we start over from `0`. Note that this would keep a different state value for
each different input
key if we had tuples with different values in the first field.
+### State time-to-live (TTL)
+
+A time-to-live (TTL) can be assigned to the keyed state value.
+In this case it will expire after the configured TTL
+and its stored value will be cleaned up based on the best effort.
+Depending on configuration, the expired state can become unavailable for read
access
+even if it is not cleaned up yet. In this case it behaves as if it does not
exist any more.
+
+The collection types of state support TTL on entry level:
+separate list elements and map entries expire independently.
+
+The behaviour of state with TTL firstly should be configured by building
`StateTtlConfiguration`:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build();
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build()
+{% endhighlight %}
+
+
+
+It has several options to consider.
+The first parameter of `newBuilder` method is mandatory, it is a value of
time-to-live itself.
+
+The update type configures when the time-to-live of state value is prolonged
(default `OnCreateAndWrite`):
+
+ - `StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite` - only on creation
and write access,
+ - `StateTtlConfiguration.TtlUpdateType.OnReadAndWrite` - also on read access.
+
+The state visibility configures whether the expired value is returned on read
access
+if it is not cleaned up yet (default `NeverReturnExpired`):
+
+ - `StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired` - expired
value is never returned,
+ - `StateTtlConfiguration.TtlStateVisibility.ReturnExpiredIfNotCleanedUp` -
returned if still available.
+
+The TTL can be enabled in descriptor for any type of state:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig =
StateTtlConfiguration.newBuilder(Time.seconds(1)).build();
+ValueStateDescriptor stateDescriptor = new
ValueStateDescriptor<>("text state", String.class);
+stateDescriptor.enableTimeToLive(ttlConfig);
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration.newBuilder(Time.seconds(1)).build()
+val stateDescriptor = new ValueStateDescriptor[String]("text state",
classOf[String])
+stateDescriptor.enableTimeToLive(ttlConfig)
+{% endhighlight %}
+
+
+
+**Notes:**
+
+- The state backends store the timestamp of last modification along with the
user value,
Review comment:
I think we can be more precise, the overhead in RocksDB is 8 byte per value
(for list/map state, each list value counts individually). For heap backend,
the overhead is one wrapper object with 2 references and a primitive long (this
means slightly different overheads for 32 and 64 bit JVMs).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add public user documentation for TTL feature
> -
>
> Key: FLINK-9637
> URL: https://issues.apache.org/jira/browse/FLINK-9637
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
>Affects Versions: 1.6.0
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[
https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561297#comment-16561297
]
ASF GitHub Bot commented on FLINK-9637:
---
eliaslevy commented on a change in pull request #6379: [FLINK-9637] Add public
user documentation for state TTL feature
URL: https://github.com/apache/flink/pull/6379#discussion_r205991168
##
File path: docs/dev/stream/state/state.md
##
@@ -266,6 +266,92 @@ a `ValueState`. Once the count reaches 2 it will emit the
average and clear the
we start over from `0`. Note that this would keep a different state value for
each different input
key if we had tuples with different values in the first field.
+### State time-to-live (TTL)
+
+A time-to-live (TTL) can be assigned to the keyed state value.
+In this case it will expire after the configured TTL
+and its stored value will be cleaned up based on the best effort.
+Depending on configuration, the expired state can become unavailable for read
access
+even if it is not cleaned up yet. In this case it behaves as if it does not
exist any more.
+
+The collection types of state support TTL on entry level:
+separate list elements and map entries expire independently.
+
+The behaviour of state with TTL firstly should be configured by building
`StateTtlConfiguration`:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build();
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build()
+{% endhighlight %}
+
+
+
+It has several options to consider.
+The first parameter of `newBuilder` method is mandatory, it is a value of
time-to-live itself.
+
+The update type configures when the time-to-live of state value is prolonged
(default `OnCreateAndWrite`):
+
+ - `StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite` - only on creation
and write access,
+ - `StateTtlConfiguration.TtlUpdateType.OnReadAndWrite` - also on read access.
+
+The state visibility configures whether the expired value is returned on read
access
+if it is not cleaned up yet (default `NeverReturnExpired`):
+
+ - `StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired` - expired
value is never returned,
+ - `StateTtlConfiguration.TtlStateVisibility.ReturnExpiredIfNotCleanedUp` -
returned if still available.
+
+The TTL can be enabled in descriptor for any type of state:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig =
StateTtlConfiguration.newBuilder(Time.seconds(1)).build();
+ValueStateDescriptor stateDescriptor = new
ValueStateDescriptor<>("text state", String.class);
+stateDescriptor.enableTimeToLive(ttlConfig);
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration.newBuilder(Time.seconds(1)).build()
+val stateDescriptor = new ValueStateDescriptor[String]("text state",
classOf[String])
+stateDescriptor.enableTimeToLive(ttlConfig)
+{% endhighlight %}
+
+
+
+**Notes:**
+
+- The state backends store the timestamp of last modification along with the
user value,
+which means that enabling this feature increases consumption of state storage.
+
+- As of current implementation the state storage is cleaned up of expired
value
+only on its explicit read access per key, e.g. calling `ValueState.value()`.
+This might change in future releases, e.g. additional strategies might be
added in background to speed up cleanup.
+
+- Only *processing time* scale is currently supported for TTL.
Review comment:
Only TTLs in reference to *processing time* are currently supported.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add public user documentation for TTL feature
> -
>
> Key: FLINK-9637
> URL: https://issues.apache.org/jira/browse/FLINK-9637
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
>Affects Versions: 1.6.0
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[
https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561296#comment-16561296
]
ASF GitHub Bot commented on FLINK-9637:
---
eliaslevy commented on a change in pull request #6379: [FLINK-9637] Add public
user documentation for state TTL feature
URL: https://github.com/apache/flink/pull/6379#discussion_r205991075
##
File path: docs/dev/stream/state/state.md
##
@@ -266,6 +266,92 @@ a `ValueState`. Once the count reaches 2 it will emit the
average and clear the
we start over from `0`. Note that this would keep a different state value for
each different input
key if we had tuples with different values in the first field.
+### State time-to-live (TTL)
+
+A time-to-live (TTL) can be assigned to the keyed state value.
+In this case it will expire after the configured TTL
+and its stored value will be cleaned up based on the best effort.
+Depending on configuration, the expired state can become unavailable for read
access
+even if it is not cleaned up yet. In this case it behaves as if it does not
exist any more.
+
+The collection types of state support TTL on entry level:
+separate list elements and map entries expire independently.
+
+The behaviour of state with TTL firstly should be configured by building
`StateTtlConfiguration`:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build();
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build()
+{% endhighlight %}
+
+
+
+It has several options to consider.
+The first parameter of `newBuilder` method is mandatory, it is a value of
time-to-live itself.
+
+The update type configures when the time-to-live of state value is prolonged
(default `OnCreateAndWrite`):
+
+ - `StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite` - only on creation
and write access,
+ - `StateTtlConfiguration.TtlUpdateType.OnReadAndWrite` - also on read access.
+
+The state visibility configures whether the expired value is returned on read
access
+if it is not cleaned up yet (default `NeverReturnExpired`):
+
+ - `StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired` - expired
value is never returned,
+ - `StateTtlConfiguration.TtlStateVisibility.ReturnExpiredIfNotCleanedUp` -
returned if still available.
+
+The TTL can be enabled in descriptor for any type of state:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig =
StateTtlConfiguration.newBuilder(Time.seconds(1)).build();
+ValueStateDescriptor stateDescriptor = new
ValueStateDescriptor<>("text state", String.class);
+stateDescriptor.enableTimeToLive(ttlConfig);
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration.newBuilder(Time.seconds(1)).build()
+val stateDescriptor = new ValueStateDescriptor[String]("text state",
classOf[String])
+stateDescriptor.enableTimeToLive(ttlConfig)
+{% endhighlight %}
+
+
+
+**Notes:**
+
+- The state backends store the timestamp of last modification along with the
user value,
+which means that enabling this feature increases consumption of state storage.
+
+- As of current implementation the state storage is cleaned up of expired
value
+only on its explicit read access per key, e.g. calling `ValueState.value()`.
+This might change in future releases, e.g. additional strategies might be
added in background to speed up cleanup.
Review comment:
In the current implementation, expired values are only removed when they are
read explicitly, e.g. by calling `ValueState.value()`. Note that this means
that under the current implementation if expired state is not read, it won't be
removed, possibly leading to ever growing state. This might change in future
releases. Additional strategies might be added that clean up expired state in
the background.
BTW, is this true given FLINK-9938?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add public user documentation for TTL feature
> -
>
> Key: FLINK-9637
> URL: https://issues.apache.org/jira/browse/FLINK-9637
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends,
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[
https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561294#comment-16561294
]
ASF GitHub Bot commented on FLINK-9637:
---
eliaslevy commented on a change in pull request #6379: [FLINK-9637] Add public
user documentation for state TTL feature
URL: https://github.com/apache/flink/pull/6379#discussion_r205990823
##
File path: docs/dev/stream/state/state.md
##
@@ -266,6 +266,92 @@ a `ValueState`. Once the count reaches 2 it will emit the
average and clear the
we start over from `0`. Note that this would keep a different state value for
each different input
key if we had tuples with different values in the first field.
+### State time-to-live (TTL)
+
+A time-to-live (TTL) can be assigned to the keyed state value.
+In this case it will expire after the configured TTL
+and its stored value will be cleaned up based on the best effort.
+Depending on configuration, the expired state can become unavailable for read
access
+even if it is not cleaned up yet. In this case it behaves as if it does not
exist any more.
+
+The collection types of state support TTL on entry level:
+separate list elements and map entries expire independently.
+
+The behaviour of state with TTL firstly should be configured by building
`StateTtlConfiguration`:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build();
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build()
+{% endhighlight %}
+
+
+
+It has several options to consider.
+The first parameter of `newBuilder` method is mandatory, it is a value of
time-to-live itself.
+
+The update type configures when the time-to-live of state value is prolonged
(default `OnCreateAndWrite`):
+
+ - `StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite` - only on creation
and write access,
+ - `StateTtlConfiguration.TtlUpdateType.OnReadAndWrite` - also on read access.
+
+The state visibility configures whether the expired value is returned on read
access
+if it is not cleaned up yet (default `NeverReturnExpired`):
+
+ - `StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired` - expired
value is never returned,
+ - `StateTtlConfiguration.TtlStateVisibility.ReturnExpiredIfNotCleanedUp` -
returned if still available.
+
+The TTL can be enabled in descriptor for any type of state:
Review comment:
TTL functionality can be enabled in the descriptor of any type of state:
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add public user documentation for TTL feature
> -
>
> Key: FLINK-9637
> URL: https://issues.apache.org/jira/browse/FLINK-9637
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
>Affects Versions: 1.6.0
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[
https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561293#comment-16561293
]
ASF GitHub Bot commented on FLINK-9637:
---
eliaslevy commented on a change in pull request #6379: [FLINK-9637] Add public
user documentation for state TTL feature
URL: https://github.com/apache/flink/pull/6379#discussion_r205990785
##
File path: docs/dev/stream/state/state.md
##
@@ -266,6 +266,92 @@ a `ValueState`. Once the count reaches 2 it will emit the
average and clear the
we start over from `0`. Note that this would keep a different state value for
each different input
key if we had tuples with different values in the first field.
+### State time-to-live (TTL)
+
+A time-to-live (TTL) can be assigned to the keyed state value.
+In this case it will expire after the configured TTL
+and its stored value will be cleaned up based on the best effort.
+Depending on configuration, the expired state can become unavailable for read
access
+even if it is not cleaned up yet. In this case it behaves as if it does not
exist any more.
+
+The collection types of state support TTL on entry level:
+separate list elements and map entries expire independently.
+
+The behaviour of state with TTL firstly should be configured by building
`StateTtlConfiguration`:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build();
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build()
+{% endhighlight %}
+
+
+
+It has several options to consider.
+The first parameter of `newBuilder` method is mandatory, it is a value of
time-to-live itself.
+
+The update type configures when the time-to-live of state value is prolonged
(default `OnCreateAndWrite`):
Review comment:
The update type configures when the state TTL is refreshed (default
`OnCreateAndWrite`):
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add public user documentation for TTL feature
> -
>
> Key: FLINK-9637
> URL: https://issues.apache.org/jira/browse/FLINK-9637
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
>Affects Versions: 1.6.0
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[
https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561292#comment-16561292
]
ASF GitHub Bot commented on FLINK-9637:
---
eliaslevy commented on a change in pull request #6379: [FLINK-9637] Add public
user documentation for state TTL feature
URL: https://github.com/apache/flink/pull/6379#discussion_r205990730
##
File path: docs/dev/stream/state/state.md
##
@@ -266,6 +266,92 @@ a `ValueState`. Once the count reaches 2 it will emit the
average and clear the
we start over from `0`. Note that this would keep a different state value for
each different input
key if we had tuples with different values in the first field.
+### State time-to-live (TTL)
+
+A time-to-live (TTL) can be assigned to the keyed state value.
+In this case it will expire after the configured TTL
+and its stored value will be cleaned up based on the best effort.
+Depending on configuration, the expired state can become unavailable for read
access
+even if it is not cleaned up yet. In this case it behaves as if it does not
exist any more.
+
+The collection types of state support TTL on entry level:
+separate list elements and map entries expire independently.
+
+The behaviour of state with TTL firstly should be configured by building
`StateTtlConfiguration`:
+
+
+
+{% highlight java %}
+StateTtlConfiguration ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build();
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val ttlConfig = StateTtlConfiguration
+.newBuilder(Time.seconds(1))
+.setTtlUpdateType(StateTtlConfiguration.TtlUpdateType.OnCreateAndWrite)
+
.setStateVisibility(StateTtlConfiguration.TtlStateVisibility.NeverReturnExpired)
+.build()
+{% endhighlight %}
+
+
+
+It has several options to consider.
+The first parameter of `newBuilder` method is mandatory, it is a value of
time-to-live itself.
Review comment:
The first parameter of `newBuilder` method is mandatory, it is the
time-to-live value.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add public user documentation for TTL feature
> -
>
> Key: FLINK-9637
> URL: https://issues.apache.org/jira/browse/FLINK-9637
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
>Affects Versions: 1.6.0
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[ https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561291#comment-16561291 ] ASF GitHub Bot commented on FLINK-9637: --- eliaslevy commented on a change in pull request #6379: [FLINK-9637] Add public user documentation for state TTL feature URL: https://github.com/apache/flink/pull/6379#discussion_r205990700 ## File path: docs/dev/stream/state/state.md ## @@ -266,6 +266,92 @@ a `ValueState`. Once the count reaches 2 it will emit the average and clear the we start over from `0`. Note that this would keep a different state value for each different input key if we had tuples with different values in the first field. +### State time-to-live (TTL) + +A time-to-live (TTL) can be assigned to the keyed state value. +In this case it will expire after the configured TTL +and its stored value will be cleaned up based on the best effort. +Depending on configuration, the expired state can become unavailable for read access +even if it is not cleaned up yet. In this case it behaves as if it does not exist any more. + +The collection types of state support TTL on entry level: +separate list elements and map entries expire independently. + +The behaviour of state with TTL firstly should be configured by building `StateTtlConfiguration`: Review comment: To use state TTL you must first build a `StateTtlConfiguration` object: This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add public user documentation for TTL feature > - > > Key: FLINK-9637 > URL: https://issues.apache.org/jira/browse/FLINK-9637 > Project: Flink > Issue Type: Sub-task > Components: State Backends, Checkpointing >Affects Versions: 1.6.0 >Reporter: Andrey Zagrebin >Assignee: Andrey Zagrebin >Priority: Major > Labels: pull-request-available > Fix For: 1.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[ https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561290#comment-16561290 ] ASF GitHub Bot commented on FLINK-9637: --- eliaslevy commented on a change in pull request #6379: [FLINK-9637] Add public user documentation for state TTL feature URL: https://github.com/apache/flink/pull/6379#discussion_r205990622 ## File path: docs/dev/stream/state/state.md ## @@ -266,6 +266,92 @@ a `ValueState`. Once the count reaches 2 it will emit the average and clear the we start over from `0`. Note that this would keep a different state value for each different input key if we had tuples with different values in the first field. +### State time-to-live (TTL) + +A time-to-live (TTL) can be assigned to the keyed state value. +In this case it will expire after the configured TTL +and its stored value will be cleaned up based on the best effort. +Depending on configuration, the expired state can become unavailable for read access +even if it is not cleaned up yet. In this case it behaves as if it does not exist any more. + +The collection types of state support TTL on entry level: +separate list elements and map entries expire independently. Review comment: The state collection types support per-entry TTLs: list elements and map entries expire independently. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add public user documentation for TTL feature > - > > Key: FLINK-9637 > URL: https://issues.apache.org/jira/browse/FLINK-9637 > Project: Flink > Issue Type: Sub-task > Components: State Backends, Checkpointing >Affects Versions: 1.6.0 >Reporter: Andrey Zagrebin >Assignee: Andrey Zagrebin >Priority: Major > Labels: pull-request-available > Fix For: 1.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[ https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561288#comment-16561288 ] ASF GitHub Bot commented on FLINK-9637: --- eliaslevy commented on a change in pull request #6379: [FLINK-9637] Add public user documentation for state TTL feature URL: https://github.com/apache/flink/pull/6379#discussion_r205990412 ## File path: docs/dev/stream/state/state.md ## @@ -266,6 +266,92 @@ a `ValueState`. Once the count reaches 2 it will emit the average and clear the we start over from `0`. Note that this would keep a different state value for each different input key if we had tuples with different values in the first field. +### State time-to-live (TTL) + +A time-to-live (TTL) can be assigned to the keyed state value. +In this case it will expire after the configured TTL +and its stored value will be cleaned up based on the best effort. Review comment: s/based on the best effort/on a best effort basis/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add public user documentation for TTL feature > - > > Key: FLINK-9637 > URL: https://issues.apache.org/jira/browse/FLINK-9637 > Project: Flink > Issue Type: Sub-task > Components: State Backends, Checkpointing >Affects Versions: 1.6.0 >Reporter: Andrey Zagrebin >Assignee: Andrey Zagrebin >Priority: Major > Labels: pull-request-available > Fix For: 1.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[ https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561289#comment-16561289 ] ASF GitHub Bot commented on FLINK-9637: --- eliaslevy commented on a change in pull request #6379: [FLINK-9637] Add public user documentation for state TTL feature URL: https://github.com/apache/flink/pull/6379#discussion_r205990514 ## File path: docs/dev/stream/state/state.md ## @@ -266,6 +266,92 @@ a `ValueState`. Once the count reaches 2 it will emit the average and clear the we start over from `0`. Note that this would keep a different state value for each different input key if we had tuples with different values in the first field. +### State time-to-live (TTL) + +A time-to-live (TTL) can be assigned to the keyed state value. +In this case it will expire after the configured TTL +and its stored value will be cleaned up based on the best effort. +Depending on configuration, the expired state can become unavailable for read access +even if it is not cleaned up yet. In this case it behaves as if it does not exist any more. Review comment: s/even if it is not cleaned up yet/even if it has yet to be removed/ s/In this case it behaves as if it does not exist any more/In that case, it behaves as if it no longer exists/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add public user documentation for TTL feature > - > > Key: FLINK-9637 > URL: https://issues.apache.org/jira/browse/FLINK-9637 > Project: Flink > Issue Type: Sub-task > Components: State Backends, Checkpointing >Affects Versions: 1.6.0 >Reporter: Andrey Zagrebin >Assignee: Andrey Zagrebin >Priority: Major > Labels: pull-request-available > Fix For: 1.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9637) Add public user documentation for TTL feature
[ https://issues.apache.org/jira/browse/FLINK-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550922#comment-16550922 ] ASF GitHub Bot commented on FLINK-9637: --- GitHub user azagrebin opened a pull request: https://github.com/apache/flink/pull/6379 [FLINK-9637] Add public user documentation for state TTL feature ## What is the purpose of the change Public user documentation for first iteration of state TTL feature ## Brief change log add TTL section in docs/dev/stream/state/state.md ## Verifying this change check TTL doc section ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (docs) You can merge this pull request into a Git repository by running: $ git pull https://github.com/azagrebin/flink FLINK-9637 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/6379.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6379 commit 792ee6b75b8ba029dc040aa019bc6526dac7253b Author: Andrey Zagrebin Date: 2018-07-20T12:57:08Z [FLINK-9637] Add public user documentation for state TTL feature > Add public user documentation for TTL feature > - > > Key: FLINK-9637 > URL: https://issues.apache.org/jira/browse/FLINK-9637 > Project: Flink > Issue Type: Sub-task > Components: State Backends, Checkpointing >Affects Versions: 1.6.0 >Reporter: Andrey Zagrebin >Assignee: Andrey Zagrebin >Priority: Major > Labels: pull-request-available > Fix For: 1.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
