[
https://issues.apache.org/jira/browse/FLUME-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15551480#comment-15551480
]
Attila Simon commented on FLUME-2716:
-------------------------------------
My previous comment might have been confusing so let me clarify:
Following the source code in EventQueueBackingStoreFile#86-103 :
{noformat}
long totalBytes = (capacity + HEADER_SIZE) * Serialization.SIZE_OF_LONG; //the
formula I wrote above
...
allocate(checkpointFile, totalBytes); //allocate the disk space, essentially
create a file with size of totalBytes, no upper limit
...
mappedBuffer = checkpointFileHandle.getChannel().map(MapMode.READ_WRITE, 0,
checkpointFile.length()); // checkpointFile.length() == totalBytes and
map function has an Integer.MAX_VALUE limit on the size of mapped content
{noformat}
>From that limitation we have a constraint:
Integer.MAX_VALUE >= checkpointFile.length()
Integer.MAX_VALUE >= totalBytes
Integer.MAX_VALUE >= (capacity + HEADER_SIZE) * Serialization.SIZE_OF_LONG
floor(Integer.MAX_VALUE/Serialization.SIZE_OF_LONG) - HEADER_SIZE >= capacity
268434426 >= capacity
I recommend to add an extra(additional to the long cast) check to
EventQueueBackingStoreFile before the allocation which checks the totalsize <
integer.max_value and if fails then print out the formula similarly to the
message constructed on lines EventQueueBackingStoreFile#96-100 as well as
throwing an exception to prevent channel startup. It has to be before the
allocation as otherwise flume would allocate a system resource(file) which
couldn't be used afterwards thus resulting a resource leak.
To relax this limitation the way how memory mapping of the checkpoint file was
requested from java.nio.channels.FileChannel has to be reconsidered, eg mapping
only different parts of same checkpoint file by multiple map calls or split the
checkpoints to multiple files are alternative solutions to this problem.
> File Channel cannot handle capacity Integer.MAX_VALUE
> -----------------------------------------------------
>
> Key: FLUME-2716
> URL: https://issues.apache.org/jira/browse/FLUME-2716
> Project: Flume
> Issue Type: Bug
> Components: Channel, File Channel
> Affects Versions: v1.6.0, v1.7.0
> Reporter: Dong Zhao
> Labels: unit-test-missing
> Fix For: v1.7.0
>
> Attachments: FLUME-2716.patch
>
>
> if capacity is set to Integer.MAX_VALUE(2147483647), checkpoint file size is
> calculated wrongly to 8224. The calculation should first cast int to long,
> then calculate the totalBytes. See the patch for details. Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
