GitHub user blbradley opened a pull request:
https://github.com/apache/kafka/pull/3083
KAFKA-1955: [WIP] Disk based buffer in Producer
Based on patch from @jkreps in [this JIRA
ticket](https://issues.apache.org/jira/browse/KAFKA-1955).
- [ ] Get some unit tests that would cover disk-backed usage
- [ ] Do some manual performance testing of this usage and understand the
impact on throughput.
- [ ] Do some manual testing of failure cases (i.e. if the broker goes down
for 30 seconds we should be able to keep taking writes) and observe how well
the producer handles the catch up time when it has a large backlog to get rid
of.
- [ ] Add a new configuration for the producer to enable this, something
like use.file.buffers=true/false.
- [ ] Add documentation that covers these new options.
I've brought the patch into sync with trunk. Testing is next, which I've
started on. I am flexible on how this can be implemented.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/blbradley/kafka kafka-disk-buffer
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/3083.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3083
----
commit 6b29fc95c394283ff4f2410ad37f7c8fcbd0d8d7
Author: Brandon Bradley <bradleytas...@gmail.com>
Date: 2017-05-17T17:12:53Z
WIP: KAFKA-1955 August 8th 2015 rebase
commit 75d2af1d7f8dda4e2fe41da60455d813d655edd0
Author: Brandon Bradley <bradleytas...@gmail.com>
Date: 2017-05-17T22:43:47Z
Merge branch 'trunk' into kafka-disk-buffer
patch works against trunk test suite
commit d3c765db789eef2fe71eca7a45dbca72e356f346
Author: Brandon Bradley <bradleytas...@gmail.com>
Date: 2017-05-17T23:14:34Z
fix imports, add whitespace from diff
commit b58118c6413a5e900f5c1ebee112bd24e8d4b119
Author: Brandon Bradley <bradleytas...@gmail.com>
Date: 2017-05-17T23:34:04Z
simple file buffer test
commit cd389f073eca18effa6449d9934aea0f90e84139
Author: Brandon Bradley <bradleytas...@gmail.com>
Date: 2017-05-17T23:35:21Z
failing unallocated memory check
commit 49b6860e6c3be4bac62937dc835d5b6f97c7ff11
Author: Brandon Bradley <bradleytas...@gmail.com>
Date: 2017-05-18T00:35:29Z
allocate buffer dynamically, passing tests
commit ed7aab5357fe9d7805dcb305d0318fb4ea770550
Author: Brandon Bradley <bradleytas...@gmail.com>
Date: 2017-05-18T00:46:10Z
failing allocated memory check
commit 875ac83096199e35307a7ef47772907607aba1f1
Author: Brandon Bradley <bradleytas...@gmail.com>
Date: 2017-05-18T00:56:47Z
do not add to free list during allocation
commit 4223e14896f4609d5bef80e97ee6d9982d2127a5
Author: Brandon Bradley <bradleytas...@gmail.com>
Date: 2017-05-18T01:20:46Z
add license
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
