[
https://issues.apache.org/jira/browse/NUTCH-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gerard Bouchar updated NUTCH-2575:
----------------------------------
Description:
There is a bug in HttpResponse::readChunkedContent that prevents it to stop
reading content when it exceeds the maximum allowed size.
There [is a variable
contentBytesRead|https://github.com/apache/nutch/blob/master/src/plugin/protocol-http/src/java/org/apache/nutch/protocol/http/HttpResponse.java#L404]
that is used to check how much content has been read, but it is never updated,
so it always stays null, and [the size
check|https://github.com/apache/nutch/blob/master/src/plugin/protocol-http/src/java/org/apache/nutch/protocol/http/HttpResponse.java#L440-L442]
always returns false (unless a single chunk is larger than the maximum allowed
content size).
This allows any server to cause out-of-memory errors on our size.
was:
There is a bug in HttpResponse::readChunkedContent that prevents it to stop
reading content when it exceeds the maximum allowed size.
There [is a variable
contentBytesRead|https://github.com/apache/nutch/blob/master/src/plugin/protocol-http/src/java/org/apache/nutch/protocol/http/HttpResponse.java#L404]
that is used to check how much content has been read, but it is never updated,
so it always stays null, and [the size
check|https://github.com/apache/nutch/blob/master/src/plugin/protocol-http/src/java/org/apache/nutch/protocol/http/HttpResponse.java#L440-L442]
always returns false (unless a single chunk is larger than the maximum allowed
content size).
> protocol-http does not respect the maximum content-size
> -------------------------------------------------------
>
> Key: NUTCH-2575
> URL: https://issues.apache.org/jira/browse/NUTCH-2575
> Project: Nutch
> Issue Type: Sub-task
> Reporter: Gerard Bouchar
> Priority: Critical
>
> There is a bug in HttpResponse::readChunkedContent that prevents it to stop
> reading content when it exceeds the maximum allowed size.
> There [is a variable
> contentBytesRead|https://github.com/apache/nutch/blob/master/src/plugin/protocol-http/src/java/org/apache/nutch/protocol/http/HttpResponse.java#L404]
> that is used to check how much content has been read, but it is never
> updated, so it always stays null, and [the size
> check|https://github.com/apache/nutch/blob/master/src/plugin/protocol-http/src/java/org/apache/nutch/protocol/http/HttpResponse.java#L440-L442]
> always returns false (unless a single chunk is larger than the maximum
> allowed content size).
> This allows any server to cause out-of-memory errors on our size.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
