[GitHub] kafka pull request #3083: KAFKA-1955: [WIP] Disk based buffer in Producer

[blbradley]

Github user blbradley closed the pull request at:

https://github.com/apache/kafka/pull/3083


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #3083: KAFKA-1955: [WIP] Disk based buffer in Producer

[blbradley]

GitHub user blbradley opened a pull request:

https://github.com/apache/kafka/pull/3083

KAFKA-1955: [WIP] Disk based buffer in Producer

Based on patch from @jkreps in [this JIRA 
ticket](https://issues.apache.org/jira/browse/KAFKA-1955).

- [ ] Get some unit tests that would cover disk-backed usage
- [ ] Do some manual performance testing of this usage and understand the 
impact on throughput.
- [ ] Do some manual testing of failure cases (i.e. if the broker goes down 
for 30 seconds we should be able to keep taking writes) and observe how well 
the producer handles the catch up time when it has a large backlog to get rid 
of.
- [ ] Add a new configuration for the producer to enable this, something 
like use.file.buffers=true/false.
- [ ] Add documentation that covers these new options.

I've brought the patch into sync with trunk. Testing is next, which I've 
started on. I am flexible on how this can be implemented.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/blbradley/kafka kafka-disk-buffer

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3083.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3083


commit 6b29fc95c394283ff4f2410ad37f7c8fcbd0d8d7
Author: Brandon Bradley 
Date:   2017-05-17T17:12:53Z

WIP: KAFKA-1955 August 8th 2015 rebase

commit 75d2af1d7f8dda4e2fe41da60455d813d655edd0
Author: Brandon Bradley 
Date:   2017-05-17T22:43:47Z

Merge branch 'trunk' into kafka-disk-buffer

patch works against trunk test suite

commit d3c765db789eef2fe71eca7a45dbca72e356f346
Author: Brandon Bradley 
Date:   2017-05-17T23:14:34Z

fix imports, add whitespace from diff

commit b58118c6413a5e900f5c1ebee112bd24e8d4b119
Author: Brandon Bradley 
Date:   2017-05-17T23:34:04Z

simple file buffer test

commit cd389f073eca18effa6449d9934aea0f90e84139
Author: Brandon Bradley 
Date:   2017-05-17T23:35:21Z

failing unallocated memory check

commit 49b6860e6c3be4bac62937dc835d5b6f97c7ff11
Author: Brandon Bradley 
Date:   2017-05-18T00:35:29Z

allocate buffer dynamically, passing tests

commit ed7aab5357fe9d7805dcb305d0318fb4ea770550
Author: Brandon Bradley 
Date:   2017-05-18T00:46:10Z

failing allocated memory check

commit 875ac83096199e35307a7ef47772907607aba1f1
Author: Brandon Bradley 
Date:   2017-05-18T00:56:47Z

do not add to free list during allocation

commit 4223e14896f4609d5bef80e97ee6d9982d2127a5
Author: Brandon Bradley 
Date:   2017-05-18T01:20:46Z

add license




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---