[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165282440 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/DefaultOperatorStateBackend.java --- @@ -325,8 +325,7 @@ public OperatorStateHandl

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348167#comment-16348167 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[GitHub] flink issue #5395: [FLINK-8308] Remove explicit yajl-ruby dependency, update...

2018-02-01 Thread StevenLangbroek
Github user StevenLangbroek commented on the issue: https://github.com/apache/flink/pull/5395 [Ruby 2.0.0 support status](https://www.ruby-lang.org/en/news/2016/02/24/support-plan-of-ruby-2-0-0-and-2-1/) > As it has been announced before, all support for Ruby 2.0.0 has ended

[jira] [Commented] (FLINK-8308) Update yajl-ruby dependency to 1.3.1 or higher

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348251#comment-16348251 ] ASF GitHub Bot commented on FLINK-8308: --- Github user StevenLangbroek commented on th

[GitHub] flink pull request #5397: [FLINK-7856][flip6] WIP JobVertexBackPressureHandl...

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5397#discussion_r165291802 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMaster.java --- @@ -867,6 +889,25 @@ public void heartbeatFromResourceManag

[GitHub] flink pull request #5397: [FLINK-7856][flip6] WIP JobVertexBackPressureHandl...

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5397#discussion_r165290983 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java --- @@ -358,6 +361,18 @@ public void start() throws Exception

[GitHub] flink pull request #5395: [FLINK-8308] Remove explicit yajl-ruby dependency,...

2018-02-01 Thread uce
Github user uce commented on a diff in the pull request: https://github.com/apache/flink/pull/5395#discussion_r165295535 --- Diff: docs/_config.yml --- @@ -77,12 +77,7 @@ defaults: layout: plain nav-pos: 9 # Move to end if no pos specified -markdo

[jira] [Commented] (FLINK-8308) Update yajl-ruby dependency to 1.3.1 or higher

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348258#comment-16348258 ] ASF GitHub Bot commented on FLINK-8308: --- Github user uce commented on a diff in the

[GitHub] flink pull request #5397: [FLINK-7856][flip6] WIP JobVertexBackPressureHandl...

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5397#discussion_r165292444 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/JobVertexBackPressureInfo.java --- @@ -0,0 +1,184 @@ +/* + * Licen

[GitHub] flink pull request #5397: [FLINK-7856][flip6] WIP JobVertexBackPressureHandl...

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5397#discussion_r165294395 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java --- @@ -305,6 +311,84 @@ public void postStop() throws Ex

[GitHub] flink pull request #5397: [FLINK-7856][flip6] WIP JobVertexBackPressureHandl...

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5397#discussion_r165291604 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMaster.java --- @@ -313,6 +325,13 @@ public JobMaster(

[jira] [Commented] (FLINK-7856) Port JobVertexBackPressureHandler to REST endpoint

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348260#comment-16348260 ] ASF GitHub Bot commented on FLINK-7856: --- Github user tillrohrmann commented on a dif

[jira] [Commented] (FLINK-7856) Port JobVertexBackPressureHandler to REST endpoint

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348261#comment-16348261 ] ASF GitHub Bot commented on FLINK-7856: --- Github user tillrohrmann commented on a dif

[jira] [Commented] (FLINK-7856) Port JobVertexBackPressureHandler to REST endpoint

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348262#comment-16348262 ] ASF GitHub Bot commented on FLINK-7856: --- Github user tillrohrmann commented on a dif

[jira] [Commented] (FLINK-7856) Port JobVertexBackPressureHandler to REST endpoint

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348259#comment-16348259 ] ASF GitHub Bot commented on FLINK-7856: --- Github user tillrohrmann commented on a dif

[jira] [Commented] (FLINK-7856) Port JobVertexBackPressureHandler to REST endpoint

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348263#comment-16348263 ] ASF GitHub Bot commented on FLINK-7856: --- Github user tillrohrmann commented on a dif

[GitHub] flink pull request #5397: [FLINK-7856][flip6] WIP JobVertexBackPressureHandl...

2018-02-01 Thread GJL
Github user GJL commented on a diff in the pull request: https://github.com/apache/flink/pull/5397#discussion_r165296332 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java --- @@ -358,6 +361,18 @@ public void start() throws Exception {

[jira] [Commented] (FLINK-7856) Port JobVertexBackPressureHandler to REST endpoint

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348267#comment-16348267 ] ASF GitHub Bot commented on FLINK-7856: --- Github user GJL commented on a diff in the

[jira] [Commented] (FLINK-7856) Port JobVertexBackPressureHandler to REST endpoint

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348269#comment-16348269 ] ASF GitHub Bot commented on FLINK-7856: --- Github user GJL commented on a diff in the

[GitHub] flink pull request #5397: [FLINK-7856][flip6] WIP JobVertexBackPressureHandl...

2018-02-01 Thread GJL
Github user GJL commented on a diff in the pull request: https://github.com/apache/flink/pull/5397#discussion_r165296204 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/JobVertexBackPressureInfo.java --- @@ -0,0 +1,184 @@ +/* + * Licensed to th

[GitHub] flink pull request #5395: [FLINK-8308] Remove explicit yajl-ruby dependency,...

2018-02-01 Thread StevenLangbroek
Github user StevenLangbroek commented on a diff in the pull request: https://github.com/apache/flink/pull/5395#discussion_r165297130 --- Diff: docs/_config.yml --- @@ -77,12 +77,7 @@ defaults: layout: plain nav-pos: 9 # Move to end if no pos specified

[jira] [Commented] (FLINK-8308) Update yajl-ruby dependency to 1.3.1 or higher

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348274#comment-16348274 ] ASF GitHub Bot commented on FLINK-8308: --- Github user StevenLangbroek commented on a

[GitHub] flink pull request #5397: [FLINK-7856][flip6] WIP JobVertexBackPressureHandl...

2018-02-01 Thread GJL
Github user GJL commented on a diff in the pull request: https://github.com/apache/flink/pull/5397#discussion_r165298568 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java --- @@ -305,6 +311,84 @@ public void postStop() throws Exception {

[jira] [Commented] (FLINK-7856) Port JobVertexBackPressureHandler to REST endpoint

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348280#comment-16348280 ] ASF GitHub Bot commented on FLINK-7856: --- Github user GJL commented on a diff in the

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165301094 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskStateManager.java --- @@ -47,10 +47,11 @@ * @param checkpointMetrics

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348292#comment-16348292 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165307761 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/state/OperatorStateBackendTest.java --- @@ -355,10 +355,11 @@ public void testSnapshotEm

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348331#comment-16348331 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165308534 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/state/StateBackendTestBase.java --- @@ -3404,12 +3405,16 @@ public String fold(String ac

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165308709 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/state/StateBackendTestBase.java --- @@ -3404,12 +3405,16 @@ public String fold(String ac

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348333#comment-16348333 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348334#comment-16348334 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165309220 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/state/StateBackendTestBase.java --- @@ -3404,12 +3405,16 @@ public String fold(String ac

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348336#comment-16348336 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165309670 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/OperatorSnapshotFinalizer.java --- @@ -0,0 +1,83 @@ +/* +

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348339#comment-16348339 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348340#comment-16348340 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165309957 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/OperatorSnapshotFinalizer.java --- @@ -0,0 +1,83 @@ +/* +

[jira] [Comment Edited] (FLINK-5479) Per-partition watermarks in FlinkKafkaConsumer should consider idle partitions

2018-02-01 Thread yanxiaobin (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16343115#comment-16343115 ] yanxiaobin edited comment on FLINK-5479 at 2/1/18 10:30 AM: I'

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165315625 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java --- @@ -859,56 +861,77 @@ public void run() {

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348362#comment-16348362 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165315816 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/OperatorSnapshotFuturesTest.java --- @@ -46,20 +47,28 @@ public v

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348364#comment-16348364 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[GitHub] flink issue #5337: [FLINK-8484][flink-kinesis-connector] Ensure a Kinesis co...

2018-02-01 Thread pluppens
Github user pluppens commented on the issue: https://github.com/apache/flink/pull/5337 Regarding the remark from @StephanEwen: perhaps it would be ok to re-use the `KinesisProxy` to return a list of all shards and compare them to the `sequenceNumsToRestore` to prune any shards that no

[jira] [Commented] (FLINK-8484) Kinesis consumer re-reads closed shards on job restart

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348367#comment-16348367 ] ASF GitHub Bot commented on FLINK-8484: --- Github user pluppens commented on the issue

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165317571 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java --- @@ -916,57 +916,78 @@ private void handleExec

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348372#comment-16348372 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[GitHub] flink issue #5337: [FLINK-8484][flink-kinesis-connector] Ensure a Kinesis co...

2018-02-01 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5337 @pluppens My only concern is that scanning the whole list of shards can be very limited to AWS Kinesis's API invoke rate limitations. Also, we would then only be cleaning up the state on restor

[jira] [Commented] (FLINK-8484) Kinesis consumer re-reads closed shards on job restart

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348374#comment-16348374 ] ASF GitHub Bot commented on FLINK-8484: --- Github user tzulitai commented on the issue

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165318246 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/StreamTaskTerminationTest.java --- @@ -227,21 +229,13 @@ protecte

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165318377 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/StreamTaskTerminationTest.java --- @@ -227,21 +229,13 @@ protecte

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348375#comment-16348375 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348376#comment-16348376 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[GitHub] flink pull request #5396: [FLINK-5820] [state backends] Split shared/exclusi...

2018-02-01 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5396#discussion_r165295768 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointProperties.java --- @@ -18,45 +18,50 @@ package org.apache.f

[GitHub] flink pull request #5396: [FLINK-5820] [state backends] Split shared/exclusi...

2018-02-01 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5396#discussion_r165318529 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/FsCheckpointStateOutputStreamTest.java --- @@ -332,6 +335,43 @@ public void

[GitHub] flink pull request #5396: [FLINK-5820] [state backends] Split shared/exclusi...

2018-02-01 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5396#discussion_r165317663 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/CompletedCheckpointTest.java --- @@ -19,58 +19,53 @@ package org.apache.flin

[jira] [Commented] (FLINK-5820) Extend State Backend Abstraction to support Global Cleanup Hooks

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-5820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348382#comment-16348382 ] ASF GitHub Bot commented on FLINK-5820: --- Github user aljoscha commented on a diff in

[jira] [Commented] (FLINK-5820) Extend State Backend Abstraction to support Global Cleanup Hooks

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-5820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348384#comment-16348384 ] ASF GitHub Bot commented on FLINK-5820: --- Github user aljoscha commented on a diff in

[jira] [Commented] (FLINK-5820) Extend State Backend Abstraction to support Global Cleanup Hooks

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-5820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348381#comment-16348381 ] ASF GitHub Bot commented on FLINK-5820: --- Github user aljoscha commented on a diff in

[GitHub] flink pull request #5396: [FLINK-5820] [state backends] Split shared/exclusi...

2018-02-01 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5396#discussion_r165295667 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointProperties.java --- @@ -18,45 +18,50 @@ package org.apache.f

[jira] [Commented] (FLINK-5820) Extend State Backend Abstraction to support Global Cleanup Hooks

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-5820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348383#comment-16348383 ] ASF GitHub Bot commented on FLINK-5820: --- Github user aljoscha commented on a diff in

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165321013 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/state/TestTaskStateManager.java --- @@ -67,8 +69,11 @@ public TestTaskStateManager(

[jira] [Created] (FLINK-8542) Do not indefinitely store closed shard's state in the FlinkKinesisConsumer

2018-02-01 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-8542: -- Summary: Do not indefinitely store closed shard's state in the FlinkKinesisConsumer Key: FLINK-8542 URL: https://issues.apache.org/jira/browse/FLINK-8542

[GitHub] flink issue #5337: [FLINK-8484][flink-kinesis-connector] Ensure a Kinesis co...

2018-02-01 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5337 Here it is: https://issues.apache.org/jira/browse/FLINK-8542 ---

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348392#comment-16348392 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[jira] [Commented] (FLINK-8484) Kinesis consumer re-reads closed shards on job restart

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348395#comment-16348395 ] ASF GitHub Bot commented on FLINK-8484: --- Github user tzulitai commented on the issue

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165322528 --- Diff: flink-core/src/test/java/org/apache/flink/util/MethodForwardingTestUtil.java --- @@ -0,0 +1,146 @@ +/* + * Licensed to the Apache Soft

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348399#comment-16348399 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165323575 --- Diff: flink-core/src/test/java/org/apache/flink/util/MethodForwardingTestUtil.java --- @@ -0,0 +1,146 @@ +/* + * Licensed to the Apache Soft

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348406#comment-16348406 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165323823 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/StateObjectCollectionTest.java --- @@ -0,0 +1,64 @@ +/* + * Licensed

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348405#comment-16348405 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165323982 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/LocalStateForwardingTest.java --- @@ -0,0 +1,213 @@ +/* +

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348408#comment-16348408 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[GitHub] flink issue #5345: [FLINK-8242] [orc] Fix predicate push-down of OrcTableSou...

2018-02-01 Thread twalthr
Github user twalthr commented on the issue: https://github.com/apache/flink/pull/5345 Thank you @fhueske. The code looks good now. I will merge this... ---

[jira] [Commented] (FLINK-8242) ClassCastException in OrcTableSource.toOrcPredicate

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348411#comment-16348411 ] ASF GitHub Bot commented on FLINK-8242: --- Github user twalthr commented on the issue:

[GitHub] flink issue #5337: [FLINK-8484][flink-kinesis-connector] Ensure a Kinesis co...

2018-02-01 Thread pluppens
Github user pluppens commented on the issue: https://github.com/apache/flink/pull/5337 Good point. An ugly workaround would be to store a timestamp when the ending number is being set on a shard, and provide a configurable/sufficiently enough (eg. 7 days) window. It would exclude the

[jira] [Commented] (FLINK-8484) Kinesis consumer re-reads closed shards on job restart

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348414#comment-16348414 ] ASF GitHub Bot commented on FLINK-8484: --- Github user pluppens commented on the issue

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165325793 --- Diff: flink-core/src/test/java/org/apache/flink/util/MethodForwardingTestUtil.java --- @@ -0,0 +1,146 @@ +/* + * Licensed to the Apache Soft

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348422#comment-16348422 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165326694 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/LocalStateForwardingTest.java --- @@ -0,0 +1,213 @@ +/* +

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348424#comment-16348424 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165329369 --- Diff: flink-contrib/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackend.java --- @@ -104,10 +107,

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165329478 --- Diff: flink-contrib/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackend.java --- @@ -104,10 +107,

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348442#comment-16348442 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348444#comment-16348444 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165331645 --- Diff: flink-contrib/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackend.java --- @@ -686,4 +714,5

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165331598 --- Diff: flink-contrib/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackend.java --- @@ -686,4 +714,5

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348463#comment-16348463 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348461#comment-16348461 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165332129 --- Diff: flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java --- @@ -316,6 +316,11 @@ */ public static final

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348467#comment-16348467 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165340082 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStore.java --- @@ -46,26 +52,63 @@ /** */ private fi

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348492#comment-16348492 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165340246 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskStateManager.java --- @@ -60,4 +62,9 @@ void reportTaskStateSnapshots(

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348494#comment-16348494 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165340438 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/filesystem/FsStateBackend.java --- @@ -501,4 +529,53 @@ public String toString() {

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348495#comment-16348495 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165341071 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskManagerServices.java --- @@ -197,7 +201,13 @@ public static TaskManager

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348501#comment-16348501 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a dif

  1   2   3   >