flink git commit: [FLINK-8517] Fix missing synchronization in TaskEventDispatcher

2018-03-05 Thread srichter
Repository: flink Updated Branches: refs/heads/master 131daa28b -> d85fe58d7 [FLINK-8517] Fix missing synchronization in TaskEventDispatcher This closes #5621. Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/d85fe58d Tree

flink git commit: [FLINK-8859][checkpointing] RocksDB backend should pass WriteOption to Rocks.put() when restoring

2018-03-05 Thread srichter
Repository: flink Updated Branches: refs/heads/release-1.5 05f3fa45d -> 6b06292fb [FLINK-8859][checkpointing] RocksDB backend should pass WriteOption to Rocks.put() when restoring (cherry picked from commit 131daa2) Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://g

flink git commit: [FLINK-8859][checkpointing] RocksDB backend should pass WriteOption to Rocks.put() when restoring

2018-03-05 Thread srichter
Repository: flink Updated Branches: refs/heads/master 3c8a673f9 -> 131daa28b [FLINK-8859][checkpointing] RocksDB backend should pass WriteOption to Rocks.put() when restoring This closes #5635. Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/r

flink git commit: [FLINK-8458][config][docs] Add config of credit-based network buffers

2018-03-05 Thread srichter
Repository: flink Updated Branches: refs/heads/release-1.5 8f995e7a0 -> 05f3fa45d [FLINK-8458][config][docs] Add config of credit-based network buffers (cherry picked from commit 3c8a673) Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/as

flink git commit: [FLINK-8458][config][docs] Add config of credit-based network buffers

2018-03-05 Thread srichter
Repository: flink Updated Branches: refs/heads/master fc0001c85 -> 3c8a673f9 [FLINK-8458][config][docs] Add config of credit-based network buffers This closes #5317. Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/3c8a673

flink git commit: [FLINK-8769][flip6] do not print error causing exceptions without debugging

2018-03-05 Thread srichter
Repository: flink Updated Branches: refs/heads/release-1.5 824eb0262 -> 8f995e7a0 [FLINK-8769][flip6] do not print error causing exceptions without debugging In DispatcherRestEndpoint and TaskExecutor, there were two places where without errors (running a job inside an IDE) exceptions were log

flink git commit: [FLINK-8769][flip6] do not print error causing exceptions without debugging

2018-03-05 Thread srichter
Repository: flink Updated Branches: refs/heads/master c74d8cac2 -> fc0001c85 [FLINK-8769][flip6] do not print error causing exceptions without debugging In DispatcherRestEndpoint and TaskExecutor, there were two places where without errors (running a job inside an IDE) exceptions were logged.

flink git commit: [FLINK-8818][yarn/s3][tests] harden YarnFileStageTest upload test for eventual consistent read-after-write

2018-03-05 Thread srichter
Repository: flink Updated Branches: refs/heads/release-1.5 7c3be9149 -> 824eb0262 [FLINK-8818][yarn/s3][tests] harden YarnFileStageTest upload test for eventual consistent read-after-write In case the newly written object cannot be read (yet), we do 4 more retries to retrieve the value and wa

flink git commit: [FLINK-8818][yarn/s3][tests] harden YarnFileStageTest upload test for eventual consistent read-after-write

2018-03-05 Thread srichter
Repository: flink Updated Branches: refs/heads/master 56fef58a3 -> c74d8cac2 [FLINK-8818][yarn/s3][tests] harden YarnFileStageTest upload test for eventual consistent read-after-write In case the newly written object cannot be read (yet), we do 4 more retries to retrieve the value and wait 50

[2/2] flink git commit: [FLINK-8737][network] disallow creating a union of UnionInputGate instances

2018-02-28 Thread srichter
[FLINK-8737][network] disallow creating a union of UnionInputGate instances Recently, the pollNextBufferOrEvent() was added but not implemented but this is used in getNextBufferOrEvent() and thus any UnionInputGate containing a UnionInputGate would have failed already. There should be no use case

[1/2] flink git commit: [hotfix][network] minor improvements in UnionInputGate

2018-02-28 Thread srichter
Repository: flink Updated Branches: refs/heads/release-1.5 f14532760 -> 926566651 [hotfix][network] minor improvements in UnionInputGate (cherry picked from commit 4203557) Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/

[2/2] flink git commit: [FLINK-8737][network] disallow creating a union of UnionInputGate instances

2018-02-28 Thread srichter
[FLINK-8737][network] disallow creating a union of UnionInputGate instances Recently, the pollNextBufferOrEvent() was added but not implemented but this is used in getNextBufferOrEvent() and thus any UnionInputGate containing a UnionInputGate would have failed already. There should be no use case

[1/2] flink git commit: [hotfix][network] minor improvements in UnionInputGate

2018-02-28 Thread srichter
Repository: flink Updated Branches: refs/heads/master 6165b3db5 -> e8de53817 [hotfix][network] minor improvements in UnionInputGate Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/42035571 Tree: http://git-wip-us.apache.or

[1/2] flink git commit: [FLINK-8747][bugfix] The tag of waiting for floating buffers in RemoteInputChannel should be updated properly

2018-02-28 Thread srichter
Repository: flink Updated Branches: refs/heads/release-1.5 d5338c415 -> f14532760 [FLINK-8747][bugfix] The tag of waiting for floating buffers in RemoteInputChannel should be updated properly (cherry picked from commit 6e9e0dd) Project: http://git-wip-us.apache.org/repos/asf/flink/repo Comm

[2/2] flink git commit: [hotfix] Fix package private and comments

2018-02-28 Thread srichter
[hotfix] Fix package private and comments (cherry picked from commit 6165b3d) Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/f1453276 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/f1453276 Diff: http://git-wip-us.a

[2/2] flink git commit: [hotfix] Fix package private and comments

2018-02-28 Thread srichter
[hotfix] Fix package private and comments Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/6165b3db Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/6165b3db Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/6165b3

[1/2] flink git commit: [FLINK-8747][bugfix] The tag of waiting for floating buffers in RemoteInputChannel should be updated properly

2018-02-28 Thread srichter
Repository: flink Updated Branches: refs/heads/master b9b7416f4 -> 6165b3db5 [FLINK-8747][bugfix] The tag of waiting for floating buffers in RemoteInputChannel should be updated properly This closes #5558. Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.

[5/7] flink git commit: [FLINK-8694][runtime] Fix notifyDataAvailable race condition

2018-02-28 Thread srichter
[FLINK-8694][runtime] Fix notifyDataAvailable race condition Before there was a race condition that might resulted in igonoring some notifyDataAvailable calls. This fixes the problem by moving buffersAvailable handling to Supartitions and adds stress test for flushAlways (without this fix this t

[7/7] flink git commit: [FLINK-8750][runtime] Improve detection of no remaining data after EndOfPartitionEvent

2018-02-28 Thread srichter
[FLINK-8750][runtime] Improve detection of no remaining data after EndOfPartitionEvent Because of race condition between: 1. releasing inputChannelsWithData lock in this method and reaching this place 2. empty data notification that re-enqueues a channel we can end up with moreAvailable flag

[1/7] flink git commit: [hotfix][tests] Deduplicate code in SingleInputGateTest

2018-02-28 Thread srichter
Repository: flink Updated Branches: refs/heads/release-1.5 623e94459 -> d5338c415 [hotfix][tests] Deduplicate code in SingleInputGateTest (cherry picked from commit 67a547a) Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit

[3/7] flink git commit: [FLINK-8760][runtime] Correctly propagate moreAvailable flag through SingleInputGate

2018-02-28 Thread srichter
[FLINK-8760][runtime] Correctly propagate moreAvailable flag through SingleInputGate Previously if we SingleInputGate was re-eqnqueuing an input channel, isMoreAvailable might incorrectly return false. This might caused some dead locks. (cherry picked from commit 6c9e267) Project: http://git-

[6/7] flink git commit: [FLINK-8805][runtime] Optimize EvenSerializer.isEvent method

2018-02-28 Thread srichter
[FLINK-8805][runtime] Optimize EvenSerializer.isEvent method For example, previously if the method was used to check for EndOfPartitionEvent and the Buffer contained huge custom event, the even had to be deserialized before performing the actual check. Now we are quickly entering the correct if/e

[4/7] flink git commit: [hotfix][tests] Do not hide original exception in SuccessAfterNetworkBuffersFailureITCase

2018-02-28 Thread srichter
[hotfix][tests] Do not hide original exception in SuccessAfterNetworkBuffersFailureITCase (cherry picked from commit 2c2e189) Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/8e62f907 Tree: http://git-wip-us.apache.org/repos/

[2/7] flink git commit: [hotfix][runtime] Remove duplicated check

2018-02-28 Thread srichter
[hotfix][runtime] Remove duplicated check (cherry picked from commit 42f71f6) Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/6b7a4480 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/6b7a4480 Diff: http://git-wip-us.a

[7/7] flink git commit: [FLINK-8750][runtime] Improve detection of no remaining data after EndOfPartitionEvent

2018-02-28 Thread srichter
[FLINK-8750][runtime] Improve detection of no remaining data after EndOfPartitionEvent Because of race condition between: 1. releasing inputChannelsWithData lock in this method and reaching this place 2. empty data notification that re-enqueues a channel we can end up with moreAvailable flag

[3/7] flink git commit: [hotfix][runtime] Remove duplicated check

2018-02-28 Thread srichter
[hotfix][runtime] Remove duplicated check Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/42f71f61 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/42f71f61 Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/42f71f

[5/7] flink git commit: [FLINK-8760][runtime] Correctly propagate moreAvailable flag through SingleInputGate

2018-02-28 Thread srichter
[FLINK-8760][runtime] Correctly propagate moreAvailable flag through SingleInputGate Previously if we SingleInputGate was re-eqnqueuing an input channel, isMoreAvailable might incorrectly return false. This might caused some dead locks. Project: http://git-wip-us.apache.org/repos/asf/flink/rep

[4/7] flink git commit: [hotfix][tests] Deduplicate code in SingleInputGateTest

2018-02-28 Thread srichter
[hotfix][tests] Deduplicate code in SingleInputGateTest Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/67a547ad Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/67a547ad Diff: http://git-wip-us.apache.org/repos/asf/fli

[2/7] flink git commit: [FLINK-8694][runtime] Fix notifyDataAvailable race condition

2018-02-28 Thread srichter
[FLINK-8694][runtime] Fix notifyDataAvailable race condition Before there was a race condition that might resulted in igonoring some notifyDataAvailable calls. This fixes the problem by moving buffersAvailable handling to Supartitions and adds stress test for flushAlways (without this fix this t

[1/7] flink git commit: [FLINK-8805][runtime] Optimize EvenSerializer.isEvent method

2018-02-28 Thread srichter
Repository: flink Updated Branches: refs/heads/master af8efe92c -> b9b7416f4 [FLINK-8805][runtime] Optimize EvenSerializer.isEvent method For example, previously if the method was used to check for EndOfPartitionEvent and the Buffer contained huge custom event, the even had to be deserialized

[6/7] flink git commit: [hotfix][tests] Do not hide original exception in SuccessAfterNetworkBuffersFailureITCase

2018-02-28 Thread srichter
[hotfix][tests] Do not hide original exception in SuccessAfterNetworkBuffersFailureITCase Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/2c2e1896 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/2c2e1896 Diff: http://

flink git commit: [FLINK-8557][checkpointing] Remove illegal characters from operator description text before using it to construct the instance directory in RocksDB

2018-02-28 Thread srichter
Repository: flink Updated Branches: refs/heads/release-1.5 cf854ccbc -> 623e94459 [FLINK-8557][checkpointing] Remove illegal characters from operator description text before using it to construct the instance directory in RocksDB (cherry picked from commit 66474da) Project: http://git-wip-u

flink git commit: [FLINK-8557][checkpointing] Remove illegal characters from operator description text before using it to construct the instance directory in RocksDB

2018-02-28 Thread srichter
Repository: flink Updated Branches: refs/heads/master f57793082 -> af8efe92c [FLINK-8557][checkpointing] Remove illegal characters from operator description text before using it to construct the instance directory in RocksDB This closes #5598. Project: http://git-wip-us.apache.org/repos/asf

flink git commit: [FLINK-8777][checkpointing] Cleanup local state more eagerly in recovery

2018-02-28 Thread srichter
Repository: flink Updated Branches: refs/heads/release-1.5 302aaeb02 -> 59b607b0c [FLINK-8777][checkpointing] Cleanup local state more eagerly in recovery (cherry picked from commit 296f9ff) Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos

flink git commit: [FLINK-8777][checkpointing] Cleanup local state more eagerly in recovery

2018-02-28 Thread srichter
Repository: flink Updated Branches: refs/heads/master 6c837d738 -> 296f9ff74 [FLINK-8777][checkpointing] Cleanup local state more eagerly in recovery This closes #5578. Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/296f

flink git commit: [hotfix] Improved logging for task local recovery

2018-02-27 Thread srichter
Repository: flink Updated Branches: refs/heads/release-1.5 a13f7b234 -> 9f0295aa3 [hotfix] Improved logging for task local recovery (cherry picked from commit 56c7560) Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/9f029

flink git commit: [hotfix] Improved logging for task local recovery

2018-02-27 Thread srichter
Repository: flink Updated Branches: refs/heads/master 8d180d5fa -> 56c756040 [hotfix] Improved logging for task local recovery Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/56c75604 Tree: http://git-wip-us.apache.org/rep

[3/3] flink git commit: [FLINK-8781][scheduler] Try to reschedule failed tasks to previous allocation

2018-02-26 Thread srichter
[FLINK-8781][scheduler] Try to reschedule failed tasks to previous allocation This closes #5403. Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/d63bc75f Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/d63bc75f Diff:

[1/3] flink git commit: [FLINK-8781][scheduler] Try to reschedule failed tasks to previous allocation

2018-02-26 Thread srichter
Repository: flink Updated Branches: refs/heads/master 647c552a2 -> d63bc75ff http://git-wip-us.apache.org/repos/asf/flink/blob/d63bc75f/flink-runtime/src/test/java/org/apache/flink/runtime/jobmanager/scheduler/SchedulerSlotSharingTest.java ---

[2/3] flink git commit: [FLINK-8781][scheduler] Try to reschedule failed tasks to previous allocation

2018-02-26 Thread srichter
http://git-wip-us.apache.org/repos/asf/flink/blob/d63bc75f/flink-runtime/src/test/java/org/apache/flink/runtime/jobmanager/scheduler/ScheduleWithCoLocationHintTest.java -- diff --git a/flink-runtime/src/test/java/org/apache/flink/

[24/26] flink git commit: [hotfix] Update RocksDB version to 5.7.5

2018-02-25 Thread srichter
[hotfix] Update RocksDB version to 5.7.5 Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/a8fc3b14 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/a8fc3b14 Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/a8fc3b1

[11/26] flink git commit: [FLINK-8360][checkpointing] Implement state storage for local recovery and integrate with task lifecycle

2018-02-25 Thread srichter
http://git-wip-us.apache.org/repos/asf/flink/blob/df3e6bb7/flink-runtime/src/test/java/org/apache/flink/runtime/state/OperatorStateBackendTest.java -- diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/state/Operato

[06/26] flink git commit: [hotfix] Introduce ShutdownHookUtil to avoid code duplication

2018-02-25 Thread srichter
[hotfix] Introduce ShutdownHookUtil to avoid code duplication (Un)registering shotdown hooks for cleanups is a very common concern in Flink. Many places in the code essentially duplicate all the code for doing this. This commit introduces a utils class and deduplicates the code. Project: http://

[08/26] flink git commit: [FLINK-8360][checkpointing] Implement state storage for local recovery and integrate with task lifecycle

2018-02-25 Thread srichter
http://git-wip-us.apache.org/repos/asf/flink/blob/df3e6bb7/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/StreamTaskStateInitializerImplTest.java -- diff --git a/flink-streaming-java/src/test/java/org/

[07/26] flink git commit: [FLINK-8360][checkpointing] Implement file-based local recovery for FsStateBackend

2018-02-25 Thread srichter
[FLINK-8360][checkpointing] Implement file-based local recovery for FsStateBackend This reverts commit 8925b7c Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/e04321fa Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/

[12/26] flink git commit: [FLINK-8360][checkpointing] Implement state storage for local recovery and integrate with task lifecycle

2018-02-25 Thread srichter
http://git-wip-us.apache.org/repos/asf/flink/blob/df3e6bb7/flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/CompletedCheckpointStoreTest.java -- diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/chec

[13/26] flink git commit: [FLINK-8360][checkpointing] Implement state storage for local recovery and integrate with task lifecycle

2018-02-25 Thread srichter
http://git-wip-us.apache.org/repos/asf/flink/blob/df3e6bb7/flink-runtime/src/main/java/org/apache/flink/runtime/state/StateSnapshotContextSynchronousImpl.java -- diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/st

[25/26] flink git commit: [hotfix] RocksDB improve resource cleanup (disposal order, dispose all WriteOptions)

2018-02-25 Thread srichter
[hotfix] RocksDB improve resource cleanup (disposal order, dispose all WriteOptions) This commit ensures that all WriteOption objects are closed and that we do not create unessesary WriteOption objects for each state. Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://gi

[15/26] flink git commit: [FLINK-8360][checkpointing] Implement state storage for local recovery and integrate with task lifecycle

2018-02-25 Thread srichter
[FLINK-8360][checkpointing] Implement state storage for local recovery and integrate with task lifecycle Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/df3e6bb7 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/df3e6bb

[09/26] flink git commit: [FLINK-8360][checkpointing] Implement state storage for local recovery and integrate with task lifecycle

2018-02-25 Thread srichter
http://git-wip-us.apache.org/repos/asf/flink/blob/df3e6bb7/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackend.java -- diff --git a/flink-state-backends/flink-

[04/26] flink git commit: [hotfix] Do not track completed checkpoint ids without incremental checkpointing

2018-02-25 Thread srichter
[hotfix] Do not track completed checkpoint ids without incremental checkpointing Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/7b5c53f5 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/7b5c53f5 Diff: http://git-wip-us

[23/26] flink git commit: [hotfix] Replace use of deprecated remove calls to RocksDB with delete

2018-02-25 Thread srichter
[hotfix] Replace use of deprecated remove calls to RocksDB with delete Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/ff6662c9 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/ff6662c9 Diff: http://git-wip-us.apache.or

[02/26] flink git commit: [hotfix] Remove outdated class OperatorStateHandles and replace it with OperatorSubtaskState

2018-02-25 Thread srichter
http://git-wip-us.apache.org/repos/asf/flink/blob/617e67c2/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperatorTest.java -- diff --git a/flink-streaming-java/src/test/java/org/ap

[21/26] flink git commit: [FLINK-8360][checkpointing] Implement file-based local recovery for RocksDBStateBackend

2018-02-25 Thread srichter
http://git-wip-us.apache.org/repos/asf/flink/blob/1619fa8a/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java -- diff --git a/flink-state-backends/f

[03/26] flink git commit: [hotfix] Remove outdated class OperatorStateHandles and replace it with OperatorSubtaskState

2018-02-25 Thread srichter
[hotfix] Remove outdated class OperatorStateHandles and replace it with OperatorSubtaskState Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/617e67c2 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/617e67c2 Diff: http

[17/26] flink git commit: [FLINK-8699][checkpointing] Create deep copy of state meta data to avoid concurrency problem with checkpoints

2018-02-25 Thread srichter
[FLINK-8699][checkpointing] Create deep copy of state meta data to avoid concurrency problem with checkpoints Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/0f271161 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/0f

[14/26] flink git commit: [FLINK-8360][checkpointing] Implement state storage for local recovery and integrate with task lifecycle

2018-02-25 Thread srichter
http://git-wip-us.apache.org/repos/asf/flink/blob/df3e6bb7/flink-runtime/src/main/java/org/apache/flink/runtime/state/AbstractKeyedStateBackend.java -- diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/state/Abstra

[01/26] flink git commit: [hotfix] Rename OperatorSnapshotResult to OperatorSnapshotFutures.

2018-02-25 Thread srichter
Repository: flink Updated Branches: refs/heads/master 4e7f03e41 -> f9a583b72 [hotfix] Rename OperatorSnapshotResult to OperatorSnapshotFutures. Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/ea0d16d4 Tree: http://git-wip-

[20/26] flink git commit: [FLINK-8360][checkpointing] Implement file-based local recovery for RocksDBStateBackend

2018-02-25 Thread srichter
http://git-wip-us.apache.org/repos/asf/flink/blob/1619fa8a/flink-state-backends/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/IncrementalRocksStreamOperatorSnapshotRestoreTest.java -- diff --git

[10/26] flink git commit: [FLINK-8360][checkpointing] Implement state storage for local recovery and integrate with task lifecycle

2018-02-25 Thread srichter
http://git-wip-us.apache.org/repos/asf/flink/blob/df3e6bb7/flink-runtime/src/test/java/org/apache/flink/runtime/state/TestTaskStateManager.java -- diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/state/TestTaskSta

[16/26] flink git commit: [hotfix] Suppress emitting non-causal exceptions from closed checkpointing thread

2018-02-25 Thread srichter
[hotfix] Suppress emitting non-causal exceptions from closed checkpointing thread This avoids that an exception that is caused by closing a running snapshot is reported. With this we avoid that users get confused by their logs or that this exception could be reported before its actual cause, th

[18/26] flink git commit: [hotfix] RocksDB make default column family first

2018-02-25 Thread srichter
[hotfix] RocksDB make default column family first According to the documentation of RocksDB, the default column family should always be created first. Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/ca523fd5 Tree: http://git

[05/26] flink git commit: [hotfix] Remove some unecessary null checks in RocksDBKeyedStateBackend

2018-02-25 Thread srichter
[hotfix] Remove some unecessary null checks in RocksDBKeyedStateBackend Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/2b55ffbe Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/2b55ffbe Diff: http://git-wip-us.apache.o

[19/26] flink git commit: [FLINK-8360][checkpointing] Documentation for local recovery

2018-02-25 Thread srichter
[FLINK-8360][checkpointing] Documentation for local recovery This closes #5239. Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/32e25eb6 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/32e25eb6 Diff: http://git-wip-us

[22/26] flink git commit: [FLINK-8360][checkpointing] Implement file-based local recovery for RocksDBStateBackend

2018-02-25 Thread srichter
[FLINK-8360][checkpointing] Implement file-based local recovery for RocksDBStateBackend Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/1619fa8a Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/1619fa8a Diff: http://gi

[26/26] flink git commit: [hotfix] Clear interrupted flag in stream task cancellation

2018-02-25 Thread srichter
[hotfix] Clear interrupted flag in stream task cancellation We clear the interrupted flag before the cleanup code block of task cancellation. Otherwise, code that would like to wait until services are properly shutdown will always immediately return from calls that are supposed to be blocking wa

flink git commit: [FLINK-8679][State Backends] Ensure that RocksDBKeyedBackend.getKeys() filters keys by namespace

2018-02-23 Thread srichter
Repository: flink Updated Branches: refs/heads/master 0651876ae -> eeac022f0 [FLINK-8679][State Backends] Ensure that RocksDBKeyedBackend.getKeys() filters keys by namespace This closes #5518. Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/re

flink git commit: [FLINK-8639][State Backends] Fix always need to seek multiple times when iterator RocksDBMapState

2018-02-22 Thread srichter
Repository: flink Updated Branches: refs/heads/master e96f28bd3 -> 1e315f0dd [FLINK-8639][State Backends] Fix always need to seek multiple times when iterator RocksDBMapState This closes #5465. Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/r

flink git commit: [FLINK-8674][runtime] Improve performance of flushAlways in StreamRecordWriter

2018-02-22 Thread srichter
Repository: flink Updated Branches: refs/heads/master 3126bf522 -> a144d0f77 [FLINK-8674][runtime] Improve performance of flushAlways in StreamRecordWriter Reduce the number of data notifications in case of flushAlways = true. Instead of notifying all of the channels/subpartitions, notify onl

[1/4] flink git commit: [FLINK-8547][network] Implement CheckpointBarrierHandler not to spill data for exactly-once

2018-02-22 Thread srichter
Repository: flink Updated Branches: refs/heads/master 831349a61 -> 3126bf522 http://git-wip-us.apache.org/repos/asf/flink/blob/3126bf52/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/io/SpillingBarrierBufferTest.java ---

[2/4] flink git commit: [FLINK-8547][network] Implement CheckpointBarrierHandler not to spill data for exactly-once

2018-02-22 Thread srichter
http://git-wip-us.apache.org/repos/asf/flink/blob/3126bf52/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/io/BarrierBufferTestBase.java -- diff --git a/flink-streaming-java/src/test/java/org/apache/flink/str

[3/4] flink git commit: [FLINK-8547][network] Implement CheckpointBarrierHandler not to spill data for exactly-once

2018-02-22 Thread srichter
http://git-wip-us.apache.org/repos/asf/flink/blob/3126bf52/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/io/BarrierBufferTest.java -- diff --git a/flink-streaming-java/src/test/java/org/apache/flink/streami

[4/4] flink git commit: [FLINK-8547][network] Implement CheckpointBarrierHandler not to spill data for exactly-once

2018-02-22 Thread srichter
[FLINK-8547][network] Implement CheckpointBarrierHandler not to spill data for exactly-once This closes #5400. Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/3126bf52 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/

flink git commit: [FLINK-8657][documentation] Fix incorrect description for configuration of async snapshot for heap based backend

2018-02-19 Thread srichter
Repository: flink Updated Branches: refs/heads/master fa024726b -> 5909b5bb7 [FLINK-8657][documentation] Fix incorrect description for configuration of async snapshot for heap based backend This closes #5490. Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-

[07/35] flink git commit: [hotfix][test] Simplify RecordWriterTest

2018-02-19 Thread srichter
[hotfix][test] Simplify RecordWriterTest Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/ec7934ef Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/ec7934ef Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/ec7934e

[08/35] flink git commit: [hotfix][runtime] Refactor ResultPartition for cleaner recycle path

2018-02-19 Thread srichter
[hotfix][runtime] Refactor ResultPartition for cleaner recycle path Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/5ad84509 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/5ad84509 Diff: http://git-wip-us.apache.org/r

[29/35] flink git commit: [FLINK-8591][runtime] Pass unfinished bufferConsumers to subpartitions

2018-02-19 Thread srichter
http://git-wip-us.apache.org/repos/asf/flink/blob/5b1e127f/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/PipelinedSubpartitionTest.java -- diff --git a/flink-runtime/src/test/java/org/apache/flink/runti

[34/35] flink git commit: [FLINK-8582][runtime] Optimize BufferBuilder writes

2018-02-19 Thread srichter
[FLINK-8582][runtime] Optimize BufferBuilder writes By introducing #commit() method on critical path we reduce number of volatile writes from 2 down to 1. This improves network throughput by 20% and restores the orignal performance for high latency cases. This closes #5423. Project: http://gi

[05/35] flink git commit: [hotfix][test] Deduplicate code in LargeRecordsTest and SpanningRecordSerializationTest

2018-02-19 Thread srichter
[hotfix][test] Deduplicate code in LargeRecordsTest and SpanningRecordSerializationTest Dedupilcated code was effectively identical, but implemented in a slightly different way. Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/comm

[17/35] flink git commit: [FLINK-8583] Pass BufferConsumer to subpartitions

2018-02-19 Thread srichter
http://git-wip-us.apache.org/repos/asf/flink/blob/e9943c58/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/SpillableSubpartitionTest.java -- diff --git a/flink-runtime/src/test/java/org/apache/flink/runti

[15/35] flink git commit: [FLINK-8590][runtime] Drop addBufferConsumerToAllSubpartitions method

2018-02-19 Thread srichter
[FLINK-8590][runtime] Drop addBufferConsumerToAllSubpartitions method Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/eb96d5d2 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/eb96d5d2 Diff: http://git-wip-us.apache.org

[27/35] flink git commit: [FLINK-8588][runtime] Handle sliced buffers in RecordDeserializer

2018-02-19 Thread srichter
[FLINK-8588][runtime] Handle sliced buffers in RecordDeserializer Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/1310c725 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/1310c725 Diff: http://git-wip-us.apache.org/rep

[11/35] flink git commit: [hotfix][runtime] Deduplicate buffersInBacklog code in Pipelined and Spillable subtartitions

2018-02-19 Thread srichter
[hotfix][runtime] Deduplicate buffersInBacklog code in Pipelined and Spillable subtartitions Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/10d11d79 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/10d11d79 Diff: http

[31/35] flink git commit: [hotfix][benchmarks] Add network stack benchmarks for LocalInputChannels

2018-02-19 Thread srichter
[hotfix][benchmarks] Add network stack benchmarks for LocalInputChannels Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/08f7284a Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/08f7284a Diff: http://git-wip-us.apache.

[21/35] flink git commit: [hotfix][runtime] Simplify PipelinedSubpartition

2018-02-19 Thread srichter
[hotfix][runtime] Simplify PipelinedSubpartition notifyBuffersAvailable is a quick call that doesn't need to be executed outside of the lock Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/57228145 Tree: http://git-wip-us.ap

[02/35] flink git commit: [hotfix][tests] Do not hide original exception in Serialization tests

2018-02-19 Thread srichter
[hotfix][tests] Do not hide original exception in Serialization tests Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/058c0ed1 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/058c0ed1 Diff: http://git-wip-us.apache.org

[19/35] flink git commit: [hotfix][runtime] Simplify RecordWriter code

2018-02-19 Thread srichter
[hotfix][runtime] Simplify RecordWriter code Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/91dc1c9a Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/91dc1c9a Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/91d

[01/35] flink git commit: [hotfix][network] Invert if check in SpanningRecordSerializer to improve readability

2018-02-19 Thread srichter
Repository: flink Updated Branches: refs/heads/master 9d5a7857f -> fa024726b [hotfix][network] Invert if check in SpanningRecordSerializer to improve readability Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/c520f6bb Tr

[28/35] flink git commit: [FLINK-8589][runtime] Add polling method to InputGate

2018-02-19 Thread srichter
[FLINK-8589][runtime] Add polling method to InputGate This is a preparation for changes in data notifications, which will not be that strict as they are now. Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/98bd689a Tree: http

[30/35] flink git commit: [FLINK-8591][runtime] Pass unfinished bufferConsumers to subpartitions

2018-02-19 Thread srichter
[FLINK-8591][runtime] Pass unfinished bufferConsumers to subpartitions Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/5b1e127f Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/5b1e127f Diff: http://git-wip-us.apache.or

[33/35] flink git commit: [hotfix][tests] Remove masking original exception in StreamTaskTimerTest

2018-02-19 Thread srichter
[hotfix][tests] Remove masking original exception in StreamTaskTimerTest Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/4c38b38b Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/4c38b38b Diff: http://git-wip-us.apache.

[12/35] flink git commit: [hotfix][runtime-tests] Immediatelly fail test when one of the futures fails

2018-02-19 Thread srichter
[hotfix][runtime-tests] Immediatelly fail test when one of the futures fails Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/433e05c7 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/433e05c7 Diff: http://git-wip-us.apa

[25/35] flink git commit: [hotfix][tests] Correctly set moreAvailable flag in StreamTestSingleInputGate and handle redundant data notifications

2018-02-19 Thread srichter
[hotfix][tests] Correctly set moreAvailable flag in StreamTestSingleInputGate and handle redundant data notifications Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/78df079b Tree: http://git-wip-us.apache.org/repos/asf/flink

[06/35] flink git commit: [FLINK-8582][runtime] Introduce BufferConsumer

2018-02-19 Thread srichter
[FLINK-8582][runtime] Introduce BufferConsumer BufferConsumer will be used in the future for reading partially written MemorySegments. On flushes instead of requesting new MemorySegment BufferConsumer code will allow to continue writting to partially filled up MemmorySegment. Project: http://gi

[14/35] flink git commit: [hotfix][tests] Reduce mockito usage in StreamTaskTest

2018-02-19 Thread srichter
[hotfix][tests] Reduce mockito usage in StreamTaskTest Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/0af22bf2 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/0af22bf2 Diff: http://git-wip-us.apache.org/repos/asf/flin

[26/35] flink git commit: [FLINK-8587][runtime] Drop unused AdaptiveSpanningRecordDeserializer

2018-02-19 Thread srichter
[FLINK-8587][runtime] Drop unused AdaptiveSpanningRecordDeserializer Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/2c0f4d4a Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/2c0f4d4a Diff: http://git-wip-us.apache.org/

[35/35] flink git commit: [hotfix][runtime] Rename setNextBufferBuilder to continueWritingWithNextBufferBuilder

2018-02-19 Thread srichter
[hotfix][runtime] Rename setNextBufferBuilder to continueWritingWithNextBufferBuilder Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/fa024726 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/fa024726 Diff: http://git-

[16/35] flink git commit: [FLINK-8584] handle read-only buffers in deserializer

2018-02-19 Thread srichter
[FLINK-8584] handle read-only buffers in deserializer Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/329f0961 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/329f0961 Diff: http://git-wip-us.apache.org/repos/asf/flink

[32/35] flink git commit: [hotfix][tests] Properly clean up RescalingITCase and allow it to run in the loop

2018-02-19 Thread srichter
[hotfix][tests] Properly clean up RescalingITCase and allow it to run in the loop Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/e7d7ef12 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/e7d7ef12 Diff: http://git-wip-

[22/35] flink git commit: [hotfix][runtime] Drop unused throws IOException

2018-02-19 Thread srichter
[hotfix][runtime] Drop unused throws IOException Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/635c29d8 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/635c29d8 Diff: http://git-wip-us.apache.org/repos/asf/flink/diff

<    1   2   3   4   5   6   7   8   >