nd Crash*/
> > > > @Test
> > > > def testCreateWithInitFileSizeCrash() {
> > > > val tempDir = TestUtils.tempDir()
> > > > val seg = new LogSegment(tempDir, 40, 1, 1000, 0, SystemTime,
> > > > false, 512*1024*1024, true)
ere")
> > > seg.append(50, ms)
> > > val ms2 = messages(60, "alpha", "beta")
> > > seg.append(60, ms2)
> > > val read = seg.read(startOffset = 55, maxSize = 200, maxOffset =
> > None)
> > > assertEq
gt; >
> >> > > All issues fixed, test cases added, performance result on windows
> >> > > attached. The patch can help improve the consume performance around
> >> > > 25%~50%.
> >> > >
> >> > > Thanks, Honghai Chen
> >> > &
; make
>> > > > sure the file not exceed config.segmentSize, actually the function
>> > > > maybeRoll already make sure that.
>> > > > When try add test case for recover, blocked by the rename related
>> > > > issue, just open on
> > Thanks, Honghai Chen
> > >
> > > -----Original Message-
> > > From: Jun Rao [mailto:j...@confluent.io]
> > > Sent: Wednesday, May 6, 2015 5:39 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP 20 Enable log prealloc
Honghai Chen
> >
> > -Original Message-
> > From: Jun Rao [mailto:j...@confluent.io]
> > Sent: Wednesday, May 6, 2015 5:39 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume
> > performance
nesday, May 6, 2015 5:39 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume
> performance under windows and some old Linux file system
>
> Thanks. Could you updated the wiki? Also, commented on the jira.
>
> Jun
>
>
@kafka.apache.org
Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume
performance under windows and some old Linux file system
Thanks. Could you updated the wiki? Also, commented on the jira.
Jun
On Tue, May 5, 2015 at 12:48 AM, Honghai Chen
wrote:
> Use config.segmentSize should be
Jun Rao [mailto:j...@confluent.io]
> Sent: Tuesday, May 5, 2015 12:51 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume
> performance under windows and some old Linux file system
>
> This seems similar to what's in
> https:
option experimental to give people a chance to try it out.
> >
> >-Jay
> >
> >On Wed, Apr 22, 2015 at 7:32 PM, Honghai Chen
> >
> >wrote:
> >
> >> Hi Roshan,
> >> Use the 'auto' value maybe will break the rule and mess u
>option experimental to give people a chance to try it out.
> >
> >-Jay
> >
> >On Wed, Apr 22, 2015 at 7:32 PM, Honghai Chen
> >
> >wrote:
> >
> >> Hi Roshan,
> >> Use the 'auto' value maybe will break the rule and mess up
&
ailto:srsubraman...@linkedin.com.INVALID]
Sent: Friday, April 24, 2015 12:57 AM
To: dev@kafka.apache.org
Cc: Roshan Naik
Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume
performance under windows and some old Linux file system
+1
Some information on how this will be tested wo
/33204/diff/2/
>> >>
>> >> Thanks, Honghai Chen
>> >> http://aka.ms/kafka
>> >> http://aka.ms/manifold
>> >>
>> >> -Original Message-
>> >> From: Honghai Chen
>> >> Sent: Wednesday, April 22, 2015 11
fastmail.fm]
> Sent: Thursday, April 23, 2015 6:27 AM
> To: dev@kafka.apache.org; Roshan Naik
> Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume
> performance under windows and some old Linux file system
>
> +1 (non-binding).
>
> --
> Harsha
>
>
AM
To: dev@kafka.apache.org; Roshan Naik
Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume
performance under windows and some old Linux file system
+1 (non-binding).
--
Harsha
On April 22, 2015 at 2:52:12 PM, Roshan Naik (ros...@hortonworks.com) wrote:
I see that it is safe to keep i
rove+consume+performance+under+windows+and+some+old+Linux+
>>file+system
>> https://issues.apache.org/jira/browse/KAFKA-1646
>> https://reviews.apache.org/r/33204/diff/2/
>>
>> Thanks, Honghai Chen
>> http://aka.ms/kafka
>> http://aka.ms/manifold
>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-20+-+Enable+log+pre
>>allocate+to+improve+consume+performance+under+windows+and+some+old+Linux+
>>file+system
>> https://issues.apache.org/jira/browse/KAFKA-1646
>> https://reviews.apache.org/r/33204/
re the
> position at end of existing file.
>
> Thanks, Honghai Chen
>
> -----Original Message-
> From: Jay Kreps [mailto:jay.kr...@gmail.com]
> Sent: Wednesday, April 22, 2015 5:22 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP 20 Enable log preallocate
Original Message-
From: Honghai Chen
Sent: Wednesday, April 22, 2015 11:12 AM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP 20 Enable log preallocate to improve consume
performance under windows and some old Linux file system
Hi Sriram,
One sentence of code missed, will
make sure the
position at end of existing file.
Thanks, Honghai Chen
-Original Message-
From: Jay Kreps [mailto:jay.kr...@gmail.com]
Sent: Wednesday, April 22, 2015 5:22 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume
perform
My understanding of the patch is that clean shutdown truncates the file
back to it's true size (and reallocates it on startup). Hard crash is
handled by the normal recovery which should truncate off the empty portion
of the file.
On Tue, Apr 21, 2015 at 10:52 AM, Sriram Subramanian <
srsubraman...
Could you describe how recovery works in this mode? Say, we had a 250 MB
preallocated segment and we wrote till 50MB and crashed. Till what point
do we recover? Also, on startup, how is the append end pointer set even on
a clean shutdown? How does the FileChannel end position get set to 50 MB
inste
+1. I've tried this on Linux and it helps reduce the spikes in append (and
hence producer) latency for high throughput writes. I am not entirely sure
why but my suspicion is that in the absence of preallocation, you see
spikes writes need to happen faster than the time it takes Linux to
allocate th
I'm also +1 on this. The change is quite small and may actually help perf
on Linux as well (we've never tried this).
I have a lot of concerns on testing the various failure conditions but I
think since it will be off by default the risk is not too high.
-Jay
On Mon, Apr 20, 2015 at 6:58 PM, Hong
+1 (non-binding)
Sure, makes sense :)
Just make sure the doc for the config includes something like "If you
are using Kafka on Windows, you probably want to set it to true", so
people will know how to use it without looking for the JIRA.
On Mon, Apr 20, 2015 at 6:58 PM, Honghai Chen
wrote:
> I
I wrote a KIP for this after some discussion on KAFKA-1646.
https://issues.apache.org/jira/browse/KAFKA-1646
https://cwiki.apache.org/confluence/display/KAFKA/KIP-20+-+Enable+log+preallocate+to+improve+consume+performance+under+windows+and+some+old+Linux+file+system
The RB is here: https://reviews
26 matches
Mail list logo
