GitHub user pnowojski opened a pull request:
https://github.com/apache/flink/pull/5423
Low latency network changes
This big PR depends on #4552 and #5314. Main purpose of this change is to
increase network throughput/performance in low latency cases (benchmark results
will be posted here later). On its own, #4552 and #5314 are causing huge
performance degradation for ~1ms flushing intervals (on top of already very
poor Flink's performance in such case). This PR is fixing making throughput in
~1ms more or less similar to ~100ms flushing interval.
## Brief change log
- #4552 and #5314 dependencies
- bunch of hotfixes/prerequisiting fixes
- [FLINK-8582][runtime] Introduce BufferConsumer
- bunch of hotfixes/prerequisiting fixes
- [FLINK-8583] Pass BufferConsumer to subpartitions
- bunch of hotfixes/prerequisiting fixes
- [FLINK-8591][runtime] Pass unfinished bufferConsumers to subpartitions
- some hotfixes
This last one ([FLINK-8591]) is the one commit that actually improves the
performance by allowing sender to append a records to a memory segment, while
`PartitionRequestQueue` in Netty is busy handling/processing/flushing previous
memory segment and when it is blocked for a new credit to arrive.
## Verifying this change
This change is a trivial rework ;)
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (**yes** /
no / don't know)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pnowojski/flink buffer-consumer
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5423.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5423
----
commit c6b1f913b01db5acdd1364063b3998ccecb8317d
Author: Zhijiang <wangzhijiang999@...>
Date: 2017-09-30T06:36:19Z
[FLINK-7456][network] Implement Netty sender incoming pipeline for
credit-based
commit 7dd59f25a11b98e896c8a514f409e8ebb3127529
Author: Zhijiang <wangzhijiang999@...>
Date: 2018-01-18T15:30:34Z
[FLINK-7456][fixup] Add the related parameters in the document
commit 62e16f0f0a52b8f831f4438c6ed99b8aa72c0145
Author: Nico Kruber <nico@...>
Date: 2018-01-18T19:54:14Z
[FLINK-7456][network] revert integrating initialCredit into
unannouncedCredit
A PartitionRequest could actually be sent multiple times and each time it
must
contain the same credit (only one of those messages will actually be used).
commit 7bdcae31953339c75927b6c1292d164da52915f1
Author: Zhijiang <wangzhijiang999@...>
Date: 2018-01-17T06:15:04Z
[hotfix] Add the switch for keeping both the old mode and the new
credit-based mode
commit 2bd12a1d3c6cf51778dec0b043fa8b06fcaf06d8
Author: Nico Kruber <nico@...>
Date: 2018-01-18T16:24:59Z
fixup! [hotfix] Add the switch for keeping both the old mode and the new
credit-based mode
commit 321221d6b00d72100088a64c57b121e9571d5dc7
Author: Nico Kruber <nico@...>
Date: 2018-01-18T20:27:46Z
[FLINK-8425][network] fix SpilledSubpartitionView not protected against
concurrent release calls
commit 0ed000e12a74dfdeff925f9640c0998cdacb4d26
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2018-01-12T15:01:29Z
[hotfix][network] Invert if check in SpanningRecordSerializer to improve
readability
commit 42eb594eedc6089d1193df39119f68820391a238
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2018-01-15T11:06:31Z
[hotfix][tests] Do not hide original exception in Serialization tests
commit df89bfc7697e9d5ce2cd0fe2a334f5834f94ebc7
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2018-01-15T08:37:05Z
[hotfix][runtime] Drop one of the two clear methods in RecordSerializer
This simplifies an API a little bit
commit e753102b6ed139e18eda3221f0ff083a3f7e3610
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2018-01-16T10:08:09Z
[hotfix][tests] Deduplicate code in LargeRecordsTest
Deduplicated code was identical.
commit d24180d972ba56f53201102ac3b34242dbd08858
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2018-01-16T10:37:06Z
[hotfix][test] Deduplicate code in LargeRecordsTest and
SpanningRecordSerializationTest
Dedupilcated code was effectively identical, but implemented in a slightly
different way.
commit 2204d76231b9532d05318ecdbe4956b3d415ed19
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2018-01-04T15:42:46Z
[FLINK-8582][runtime] Introduce BufferConsumer
BufferConsumer will be used in the future for reading partially written
MemorySegments. On flushes instead of requesting new MemorySegment
BufferConsumer
code will allow to continue writting to partially filled up MemmorySegment.
commit 53905ece9f2aa2cd21ad4e9e9812270753be3be8
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2018-01-18T08:14:40Z
[hotfix][test] Simplify RecordWriterTest
commit 442980b27d9633eb4cf1031f54d30434692f35ae
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2018-01-18T09:22:23Z
[hotfix][runtime] Refactor ResultPartition for cleaner recycle path
commit 5abd37404ed39a94b3cb08e7900e57ca9c87bdfd
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2018-01-18T09:22:52Z
[hotfix][runtime] Fix recycleBuffer in ResultPartitionTest
commit 351bea462ddc1e0580c2462871354a3403586ce4
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2018-01-18T09:41:18Z
[hotfix][runtime] Deduplicate code in PipelinedSubpartition
commit 3c31567d71d81b8027ea1b9aa5b4245b6d5f8585
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2018-01-18T11:09:15Z
[hotfix][runtime] Deduplicate buffersInBacklog code in Pipelined and
Spillable subtartitions
commit 6e6ce1f7082ac5798b62896db03e149adda71121
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2018-01-23T14:34:57Z
[hotfix][runtime-tests] Immediatelly fail test when one of the futures fails
commit c1885a1c7c2a64abb8efeb244ecf896e8d701de3
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2018-01-23T16:28:13Z
[hotfix][runtime-tests] Deduplicate CollectingResultPartitionWriters classes
commit e88c0b2618abb0f168eb366bb11224ed2d8e73e1
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2018-02-07T12:34:49Z
[hotfix][tests] Reduce mockito usage in StreamTaskTest
commit 845e8607b1176f4f8bd7b445aa1c939a77b921f3
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2018-02-06T09:03:03Z
[FLINK-8590][runtime] Drop addBufferConsumerToAllSubpartitions method
commit ebe4c7c8e7fc1b6a444712c1fe3a128f01e0b30c
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2018-01-18T09:28:48Z
[FLINK-8583] Pass BufferConsumer to subpartitions
commit bca5301c438218def5fd56886bdc91bde657b6b6
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2018-01-24T14:41:22Z
[FLINK-8584] handle read-only buffers in deserializer
commit da7eb1550df71b6bdb0747d4d481b36f271749f4
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2018-01-25T14:19:35Z
[hotfix][runtime] Simplify RecordWriter code
commit 145c2fbc39547b4825d701e4115039cbc62854c4
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2018-01-25T14:31:01Z
[hotfix][java-docs] Improve ResultSubpartition java doc
commit 386ad771def518e9adc2eb56157aad8193317a7d
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2018-01-25T14:50:42Z
[hotfix][runtime] Simplify PipelinedSubpartition
notifyBuffersAvailable is a quick call that doesn't need to be executed
outside of the lock
commit d59edad1683967bba9a0cfa86bd476e0710d4876
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2018-01-25T15:23:36Z
[hotfix][runtime] Drop unused throws IOException
commit 36a448432705e5c87b3cdc5b8d97d633a46f6dc6
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2018-01-30T08:17:50Z
[FLINK-8586][tests] Clean up hard to maintain tests
SpilledSubpartitionViewTest duplicates a lot of production logic
(TestSubpartitionConsumer is a
duplicated logic of LocalInputChannel and mix of
CreditBasedSequenceNumberingViewReader with PartitionRequestQueue.
Also it seems like most of the logic is covered by
SpillableSubpartitionTest.
commit 4c28d4b8b414e3ef41aa01a070b90f33ff3eabbc
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2018-01-29T12:00:33Z
[hotfix][tests] Properly close StreamRecordWriter in network benchmarks
commit f8b334cf9e4c9b8d5a51c3a40a71304c3f03dfc4
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2018-01-30T16:03:14Z
[hotfix][tests] Correctly set moreAvailable flag in
StreamTestSingleInputGate and handle redundant data notifications
----
---
