Hi Andrea,
We're connecting to a postgres 9.1 postgis 2.0 database.
I can download as much data as I want after removing the gzip filter
(several gigabytes) .
I was looking at the revision history in subversion which I'm assuming
is correct for those earlier changes, but yes I did read that history
incorrectly.
At the risk of getting it wrong again, the changes made for GEOS-2626
were made in trunk in r11367 - see attached r11367 diff
M
/trunk/src/web/src/main/java/org/geoserver/filters/GZIPResponseStream.java
M
/trunk/src/web/src/main/java/org/geoserver/filters/GZIPResponseWrapper.java
Prior to these changes, the response was gzipped to a
ByteArrayOutputStream and when that was finished the result was copied
to the servlet output stream (i.e. the gzipped response was stored in
memory). After the change the response is gzipped directly to the
servlet output stream.
As far as I can tell, the current code has reverted to gzipping to a
ByteArrayOutputStream and then copying to the servlet output stream.
Its visible in the stack trace below.
Looking through the old repository logs it looks like the filters in
/trunk/src/web/src/main/java/org/geoserver/filters were deleted in
r11466. The current filter appears to be sourced from
src/main/src/main/java/org/geoserver/filters/GZIPResponseStream.java.
This version of the filter has been around since before the changes
above were made but the streaming changes above were never applied to it.
Regards,
CraigJ
On 09/07/13 22:19, Andrea Aime wrote:
On Tue, Jul 9, 2013 at 6:27 AM, Craig Jones <craig.jo...@utas.edu.au
<mailto:craig.jo...@utas.edu.au>> wrote:
Hi All,
We are getting the exception below when downloading a large wfs
response in csv or gml format using gzip compression. Were are
using geoserver 2.1.1 in production but this also happens in 2.3.3.
java.lang.OutOfMemoryError: Java heap space
java.util.Arrays.copyOf(Arrays.java:2271)
java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:253)
java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:211)
java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:146)
org.geoserver.filters.GZIPResponseStream.write(GZIPResponseStream.java:86)
org.geoserver.filters.GZIPResponseStream.write(GZIPResponseStream.java:79)
org.geoserver.filters.AlternativesResponseStream.write(AlternativesResponseStream.java:53)
org.geoserver.wfs.response.WfsExceptionHandler.handle1_0(WfsExceptionHandler.java:132)
org.geoserver.wfs.response.WfsExceptionHandler.handleDefault(WfsExceptionHandler.java:81)
org.geoserver.wfs.response.WfsExceptionHandler.handleServiceException(WfsExceptionHandler.java:59)
org.geoserver.ows.Dispatcher.handleServiceException(Dispatcher.java:1638)
org.geoserver.ows.Dispatcher.exception(Dispatcher.java:1583)
org.geoserver.ows.Dispatcher.handleRequestInternal(Dispatcher.java:282)
org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
From the log message above, geoserver is falling over writing all
the gzipped output to a ByteArrayOutputStream which in this case
filled up all available memory.
This is a serious issue, because it goes against the basic design
principles of GeoServer, where each data access is performed
in a streaming way, read a bit of input, encode a bit out output, repeat.
That said, can you describe a bit more your scenario? There is one
data source that unfortunately cannot do streaming (the database is
not JDBC compliant, ignores the fetch size we set), which is mysql, is
that your case?
I note that this was the subject of a previous issue GEOS-2626,
but it ooks to me like the changes made there were reverted as a
part of the changes made in GEOS-4845 r16568.
I'm confused about the reference to GEOS-4845, it just moved the
existing files from web/app to main to make it easier building a
custom version
of GeoServer without having to recur to maven war overlays?
Mind, the history in git is only 2 years deep, using "git blame" often
results in surprising or incomplete outputs
Anyways, the gzip compression filter can be disabled by manipulating
the web.xml file, have you tried removing it? Does that fix your issue?
Cheers
Andrea
--
==
Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information.
==
Ing. Andrea Aime
@geowolf
Technical Lead
GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549
http://www.geo-solutions.it
http://twitter.com/geosolutions_it
-------------------------------------------------------
Index: GZIPResponseStream.java
===================================================================
--- GZIPResponseStream.java (revision 11366)
+++ GZIPResponseStream.java (revision 11367)
@@ -1,86 +1,83 @@
-/*
- * Copyright 2003 Jayson Falkner (jay...@jspinsider.com)
- * This code is from "Servlets and JavaServer pages; the J2EE Web Tier",
- * http://www.jspbook.com. You may freely use the code both commercially
- * and non-commercially. If you like the code, please pick up a copy of
- * the book and help support the authors, development of more free code,
- * and the JSP/Servlet/J2EE community.
- *
- * Modified by David Winslow <dwins...@openplans.org>
- */
-package org.geoserver.filters;
-
-import java.io.IOException;
-import java.io.ByteArrayOutputStream;
-import java.util.zip.GZIPOutputStream;
-import javax.servlet.ServletOutputStream;
-import javax.servlet.http.HttpServletResponse;
-
-
-public class GZIPResponseStream extends ServletOutputStream {
- protected ByteArrayOutputStream baos = null;
- protected GZIPOutputStream gzipstream = null;
- protected boolean closed = false;
- protected HttpServletResponse response = null;
- protected ServletOutputStream output = null;
-
- public GZIPResponseStream(HttpServletResponse response) throws IOException {
- super();
- closed = false;
- this.response = response;
- this.output = response.getOutputStream();
- baos = new ByteArrayOutputStream();
- gzipstream = new GZIPOutputStream(baos);
- }
-
- public void close() throws IOException {
- if (closed) {
- throw new IOException("This output stream has already been closed");
- }
- gzipstream.finish();
-
- byte[] bytes = baos.toByteArray();
-
-
- response.addHeader("Content-Length",
- Integer.toString(bytes.length));
- response.addHeader("Content-Encoding", "gzip");
- output.write(bytes);
- output.flush();
- output.close();
- closed = true;
- }
-
- public void flush() throws IOException {
- if (closed) {
- throw new IOException("Cannot flush a closed output stream");
- }
- gzipstream.flush();
- }
-
- public void write(int b) throws IOException {
- if (closed) {
- throw new IOException("Cannot write to a closed output stream");
- }
- gzipstream.write((byte)b);
- }
-
- public void write(byte b[]) throws IOException {
- write(b, 0, b.length);
- }
-
- public void write(byte b[], int off, int len) throws IOException {
- if (closed) {
- throw new IOException("Cannot write to a closed output stream");
- }
- gzipstream.write(b, off, len);
- }
-
- public boolean closed() {
- return (this.closed);
- }
-
- public void reset() {
- //noop
- }
-}
+/*
+ * Copyright 2003 Jayson Falkner (jay...@jspinsider.com)
+ * This code is from "Servlets and JavaServer pages; the J2EE Web Tier",
+ * http://www.jspbook.com. You may freely use the code both commercially
+ * and non-commercially. If you like the code, please pick up a copy of
+ * the book and help support the authors, development of more free code,
+ * and the JSP/Servlet/J2EE community.
+ *
+ * Modified by David Winslow <dwins...@openplans.org>
+ */
+package org.geoserver.filters;
+
+import java.io.IOException;
+import java.io.ByteArrayOutputStream;
+import java.util.zip.GZIPOutputStream;
+import javax.servlet.ServletOutputStream;
+import javax.servlet.http.HttpServletResponse;
+
+
+public class GZIPResponseStream extends ServletOutputStream {
+ protected GZIPOutputStream gzipstream = null;
+ protected boolean closed = false;
+ protected HttpServletResponse response = null;
+ protected ServletOutputStream output = null;
+
+ public GZIPResponseStream(HttpServletResponse response) throws IOException {
+ super();
+ closed = false;
+ this.response = response;
+ response.addHeader("Content-Encoding", "gzip");
+ this.output = response.getOutputStream();
+ // create a 4kb buffered gzip output stream
+ gzipstream = new GZIPOutputStream(output, 4 * 1024);
+ }
+
+ public void close() throws IOException {
+ if (closed) {
+ throw new IOException("This output stream has already been closed");
+ }
+ gzipstream.finish();
+
+ output.flush();
+ output.close();
+ closed = true;
+ }
+
+ public void flush() throws IOException {
+ if (closed) {
+ throw new IOException("Cannot flush a closed output stream");
+ }
+ gzipstream.flush();
+ }
+
+ public void write(int b) throws IOException {
+ if (closed) {
+ throw new IOException("Cannot write to a closed output stream");
+ }
+ gzipstream.write((byte)b);
+ }
+
+ public void write(byte b[]) throws IOException {
+ if (closed) {
+ throw new IOException("Cannot write to a closed output stream");
+ }
+
+ write(b, 0, b.length);
+ }
+
+ public void write(byte b[], int off, int len) throws IOException {
+ if (closed) {
+ throw new IOException("Cannot write to a closed output stream");
+ }
+ gzipstream.write(b, off, len);
+ }
+
+ public boolean closed() {
+ return (this.closed);
+ }
+
+ public void reset() {
+ //noop
+ }
+}
Index: GZIPResponseWrapper.java
===================================================================
--- GZIPResponseWrapper.java (revision 11366)
+++ GZIPResponseWrapper.java (revision 11367)
@@ -10,14 +10,16 @@
*/
package org.geoserver.filters;
-import java.io.*;
-import java.util.*;
-import javax.servlet.*;
-import javax.servlet.http.*;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
import java.util.Set;
-import java.util.HashSet;
import java.util.logging.Logger;
+import javax.servlet.ServletOutputStream;
+import javax.servlet.http.HttpServletResponse;
+import javax.servlet.http.HttpServletResponseWrapper;
+
public class GZIPResponseWrapper extends HttpServletResponseWrapper {
protected HttpServletResponse origResponse = null;
protected AlternativesResponseStream stream = null;
@@ -39,10 +41,6 @@
}
public void setContentType(String type){
-// if (stream != null && stream.isDirty()){
-// logger.warning("Setting mimetype after acquiring stream! was:" +
-// getContentType() + "; set to: " + type + "; url was: " + requestedURL);
-// }
origResponse.setContentType(type);
}
------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users
