> On Sept. 14, 2012, 8:44 p.m., Mike Percy wrote: > > We need to find another way to do it than make it synchronized. That would > > kill multithreaded performance. > > Ted Malaska wrote: > You mean on the put? Hmm let me think about it > > Ted Malaska wrote: > How about a on the first if statement getAndAdd(+eventSize) and a > getAndAdd(-eventSize) if the event doesn't get into the queue? > > Ted Malaska wrote: > Talked with Mike > > Here is the plan for the weekend > 1. Change the put logic to not use sync but a getAndAdd(+EventSize) at > the time of if and getAndAdd(-eventSize_ if there is a failure getting into > the queue > 2. Single a log.info if config doesn't have memory limit set. > > Hari Shreedharan wrote: > Suggestion: You could use a semaphore to keep track of the bytes > used/available. A semaphore can go up only to about 1.5G(since it uses > integer). You could definite granularity limit of 10 bytes or so(1 permit on > the semaphore buys 10 bytes - so a 50 byte event requires 5 permits(+ > calculate object overheads), to increase this to 15G - which seems reasonable. > > Ted Malaska wrote: > Thanks Hari, > > I've never used Semaphore, but they look really cool. I don't know if I > like the 15G limit. hmm. Let me finish 1295 then I'm start playing with both > ideas. > >
Well, if you increase granularity to 100 bytes, you can go right upto 150G. I would say 30 bytes granularity(~45G is a pretty reasonable value). - Hari ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/6982/#review11563 ----------------------------------------------------------- On Sept. 14, 2012, 2 p.m., Ted Malaska wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/6982/ > ----------------------------------------------------------- > > (Updated Sept. 14, 2012, 2 p.m.) > > > Review request for Flume. > > > Description > ------- > > 1. The user will be able to define a byteCapacity and a > byteCapacityBufferPercentage. > 2. Events byte size will be estimated from there body contents > 3. On put bytes are added to current total > 4. On commit any uncommitted takes are removed from the current byte total > 5. On rollover any uncommitted puts are removed from the current byte total > > > This addresses bug FLUME-1535. > https://issues.apache.org/jira/browse/FLUME-1535 > > > Diffs > ----- > > flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java > c72e97c > flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java > e070864 > > Diff: https://reviews.apache.org/r/6982/diff/ > > > Testing > ------- > > > Thanks, > > Ted Malaska > >
