[
https://issues.apache.org/jira/browse/FLUME-2155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13841465#comment-13841465
]
Brock Noland commented on FLUME-2155:
-------------------------------------
Hi Hari,
In my testing this patch does look to improve the performance of copy! Nice
work!
However, I was unable to find a scenerio where copy took a significant amount
of time during replay.
I think we should find a case where copy takes a large amount of time during a
file channel replay
before considering the change to the copy code.
However, in testing this patch, I do believe I have found the scenerio I have
seen most often seen
cause long replays. In addition this is the scenerio I believe played out in
FLUME-2118. In the
thread dump attached to the ticket you see the code in FEQ.remove():
{noformat}
"lifecycleSupervisor-1-0" prio=10 tid=0x00007fea505f7000 nid=0x279e runnable
[0x00007fe84240d000]
java.lang.Thread.State: RUNNABLE
at
org.apache.flume.channel.file.FlumeEventQueue.remove(FlumeEventQueue.java:195)
- locked <0x00007fe84d0007b8> (a
org.apache.flume.channel.file.FlumeEventQueue)
at
org.apache.flume.channel.file.ReplayHandler.processCommit(ReplayHandler.java:404)
at
org.apache.flume.channel.file.ReplayHandler.replayLog(ReplayHandler.java:327)
at org.apache.flume.channel.file.Log.doReplay(Log.java:503)
at org.apache.flume.channel.file.Log.replay(Log.java:430)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:301)
{noformat}
In the attached patch, FLUME-FC-SLOW-REPLAY-1.patch which is for demonstration
purposes, there is a
new file TestFileChannelReplayPerformance.java. The test in that file
demonstrates the issue.
The issue is that is when there are many takes in files that don't have
associated put's we search
the entire queue which is O(N) for *each* take in the file.
As you noted, using an fast lookup data structure would solve this. However, if
it were a memory
based data structure it would also consume large amounts of memory where it did
not previously.
Specifically your example of 100 million capacity would result in a 1.4GB data
structure
(100 million * 16 bytes - I use 16 bytes because we have to store Long objects
and assuming 64bit JVM).
I think we need an off-heap fast data structure to perform these lookups. There
is a project
called MapDB (former jdbm) I have used in the past which provides such a data
structure.
In the attached patch, FLUME-FC-SLOW-REPLAY-FIX-1.patch, I have used it to
provide an off-heap Set
which mirrors the FEQ. Without FLUME-FC-SLOW-REPLAY-FIX-1.patch,
TestFileChannelReplayPerformance
takes 50 minutes to replay while it takes only 6.5 minutes with the fix.
Without fix:
{noformat}
FlumeEventQueue.logTimings Search Count = 669000.0, Search Time = 3044957.0,
Copy Count = 321014.0, Copy Time = 1103.0
TestFileChannelReplayPerformance.testReplayPerformanc Total Replay Time =
3500624
{noformat}
With fix:
{noformat}
FlumeEventQueue.logTimings Search Count = 274449.0, Search Time = 1080.0, Copy
Count = 274449.0, Copy Time = 1012.0
TestFileChannelReplayPerformance.testReplayPerformance Total Replay Time =
396338
{noformat}
NOTE: FLUME-FC-SLOW-REPLAY-1.patch and FLUME-FC-SLOW-REPLAY-FIX-1.patch are not
for commit.
> Improve replay time
> -------------------
>
> Key: FLUME-2155
> URL: https://issues.apache.org/jira/browse/FLUME-2155
> Project: Flume
> Issue Type: Bug
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Attachments: 10000-20000, 100000-110000, 300000-310000,
> 700000-710000, FLUME-2155-initial.patch, FLUME-2155.patch,
> FLUME-FC-SLOW-REPLAY-1.patch, FLUME-FC-SLOW-REPLAY-FIX-1.patch,
> SmartReplay.pdf, SmartReplay1.1.pdf, fc-test.patch
>
>
> File Channel has scaled so well that people now run channels with sizes in
> 100's of millions of events. Turns out, replay can be crazy slow even between
> checkpoints at this scale - because of the remove() method in FlumeEventQueue
> moving every pointer that follows the one being removed (1 remove causes 99
> million+ moves for a channel of 100 million!). There are several ways of
> improving - one being move at the end of replay - sort of like a compaction.
> Another is to use the fact that all removes happen from the top of the queue,
> so move the first "k" events out to hashset and remove from there - we can
> find k using the write id of the last checkpoint and the current one.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
