[postgis-users] ERROR: array size exceeds the maximum allowed (134217727)

2010-03-20 Thread Mike Leahy
Hello list, I have an interesting error that's coming up on an Ubuntu 64-bit system, running PostgreSQL 8.4.2, and PostGIS 1.5.1 (packaged by me) as well as 1.4.0 (from Ubuntu repos). The query itself is somewhat complex (I'm sure it could be optimized better - but it's been working well for q

Re: [postgis-users] ERROR: array size exceeds the maximum allowed (134217727)

2010-03-20 Thread Mike Leahy
Hello again, It might be of interest to point out that substituting st_union() with st_memunion() seems to have worked around this. I'm curious though, because there is not a great deal of data being processed, and I am running this on a fairly sturdy system that that has more capacity than so

Re: [postgis-users] ERROR: array size exceeds the maximum allowed (134217727)

2010-03-20 Thread strk
On Sat, Mar 20, 2010 at 05:49:42AM -0400, Mike Leahy wrote: > Hello again, > > It might be of interest to point out that substituting st_union() with > st_memunion() seems to have worked around this. I'm curious though, because > there is not a great deal of data being processed, and I am runni

Re: [postgis-users] ERROR: array size exceeds the maximum allowed (134217727)

2010-03-20 Thread Paul Ramsey
Actually, memunion does the opposite, it passes the resultant and preserves mem. The default behavior is fast-but-memory-hungry. And has been for some time, though in different forms. There were some bugs in the array handling code, but Mark CA killed most of them, so the latest 1.5 and 1.4 streams

Re: [postgis-users] ERROR: array size exceeds the maximum allowed(134217727)

2010-03-20 Thread Paragon Corporation
Paul, I doubt array size limit is the issue. He said when he left the where condition out it worked. I would think it would definitely blow up in that case. Mike, The issue from before was that array aggregate functions did not handle NULLs correctly. 64-bit systems were more likely to segfault

Re: [postgis-users] ERROR: array size exceeds the maximum allowed (134217727)

2010-03-20 Thread strk
On Sat, Mar 20, 2010 at 07:56:14AM -0700, Paul Ramsey wrote: > Actually, memunion does the opposite, it passes the resultant and > preserves mem. Oops, sorry for the confusion. Not-so-intuitive name :P --strk; () Free GIS & Flash consultant/developer /\ http://strk.keybit.net/services.ht

Re: [postgis-users] ERROR: array size exceeds the maximum allowed (134217727)

2010-03-20 Thread Chris Hermansen
Very recently there was a kernel patch in Ubuntu regarding - if memory serves - memory copying. How up to date is the system in question? strk wrote: On Sat, Mar 20, 2010 at 07:56:14AM -0700, Paul Ramsey wrote: Actually, memunion does the opposite, it passes the resultant and preserves mem

Re: [postgis-users] ERROR: array size exceeds the maximum allowed(134217727)

2010-03-20 Thread Mike Leahy
Hi Leo/Regina, Thanks for the suggestion. I tried adding the clause 'not geom is null' to the where statement in each of the two sub-queries that have the st_union(geom) functions are used, but it still segfaults. I also tried this on a fresh database with very little data, and it doesn't see

Re: [postgis-users] ERROR: array size exceeds the maximum allowed (134217727)

2010-03-20 Thread Mike Leahy
Chris, Sorry for not catching this one sooner (I'm subscribed to the digest)...but thanks for pointing this out. The system is completely up to date. I could reboot into an older kernel to see if that solve the problem. I'll report results a bit later today. Mike ---

Re: [postgis-users] ERROR: array size exceeds the maximum allowed (134217727)

2010-03-20 Thread Mike Leahy
Chris/list: I have tried booting into the following kernels, all exhibiting the same issue: 2.6.31-20-generic 2.6.31-19-generic 2.6.31-14-generic I believe 2.6.31-14-generic is the original kernel installed with this version of Ubuntu, so unless the change to the kernel you're thinking occurre

Re: [postgis-users] ERROR: array size exceeds the maximum allowed (134217727)

2010-03-20 Thread Chris Hermansen
No, the patch was very recent or I would not have remembered it. FWIW I'm also running 2.6.31-20-generic. Too bad, it would have been an easy solution! Mike Leahy wrote: Chris/list: I have tried booting into the following kernels, all exhibiting the same issue: 2.6.31-20-generic 2.6.31-19

Re: [postgis-users] ERROR: array size exceeds the maximum allowed(134217727)

2010-03-20 Thread Paragon Corporation
Mike, We apologize, didn't notice this was in a subquery and that you have a limit statement in your query. So we presume regardless of your WHERE only 26 records are being selected. So Paul could be right that you do have data that is hitting some compiled or variable limit. Does running the

Re: [postgis-users] ERROR: array size exceeds the maximum allowed(134217727)

2010-03-20 Thread Mike Leahy
Hey, The query (when it works) is virtually instantaneous when I use st_memunion() (or when I took out part of the where clause - although now I can't even get that to work, so maybe it was just lucky). Each of the subqueries also work fine without any obvious problem. To me, it seems to be a

Re: [postgis-users] ERROR: array size exceeds the maximum allowed(134217727)

2010-03-20 Thread Mike Leahy
FWIW, if I remove the entire where clause at the end of the statement, it works again (with st_union()) and is virtually instantaneous. On Saturday 20 March 2010 18:10:38 Mike Leahy wrote: > Hey, > > The query (when it works) is virtually instantaneous when I use > st_memunion() (or when I too

Re: [postgis-users] Read file encoding from from shape file

2010-03-20 Thread Peter Hopfgartner
Paul Ramsey wrote: I was never certain that ESRI software actually respected the DBF code page flags, but your reference indicates they do, so this seems reasonable. Some investigation of the current state of shapelib WRT encoding would be wise, as I think we should update our shapelib to the cur

Re: [postgis-users] ERROR: array size exceeds the maximum allowed(134217727)

2010-03-20 Thread Paul Ramsey
Mike Well, if you can reduce it to a small db dump and query that exercises it, I can install a 64bit ubuntu VM and see if it happens here too. P On Sat, Mar 20, 2010 at 3:30 PM, Mike Leahy wrote: > FWIW,  if I remove the entire where clause at the end of the statement, it > works again (with s

Re: [postgis-users] Read file encoding from from shape file

2010-03-20 Thread Paul Ramsey
Yes, there's patches to our shapelib that don't exist in the mainline. The process needs to identify our changes to feed back to mainline, update mainline, then bring that in. P. On Sat, Mar 20, 2010 at 3:41 PM, Peter Hopfgartner wrote: > Paul Ramsey wrote: >> >> I was never certain that ESRI so

Re: [postgis-users] ERROR: array size exceeds the maximumallowed(134217727)

2010-03-20 Thread Paragon Corporation
Mike, Actually scanning thru the archives that 13... is a common number so suspect that is an OS limit. This could be very well a bug in PostgreSQL 8.4. Have you tried this on other PostgreSQL 8.4 installs on other your Fedoras - you mentioned trying on older installs? Your problem sounds ex

Re: [postgis-users] ERROR: array size exceeds themaximumallowed(134217727)

2010-03-20 Thread Paragon Corporation
Mike, Another thought. I suppose it could be the more or less two identical subselects both with array_aggs Just to give Paul something hopefully shorter to work with. Does this fail too? SELECT csls.chat_id, csls.ugeom FROM ( SELECT '[stuff]' AS selection, st_union(geom) AS ugeom, csl1.ch

Re: [postgis-users] ERROR: array size exceeds themaximumallowed(134217727)

2010-03-20 Thread Mike Leahy
Hi, That query there does not crash. I'll try to follow up on your suggestions in the prior email. Mike On Saturday 20 March 2010 23:35:32 Paragon Corporation wrote: > Mike, > Another thought. I suppose it could be the more or less two identical > subselects both with array_aggs > Just to gi

Re: [postgis-users] ERROR: array size exceeds themaximumallowed(134217727)

2010-03-20 Thread Mike Leahy
Leo/Regina, In response to your earlier message, I did find that report of the identical error message. However, as I'm sure you noticed, there is no apparent investigation that followed after it. That was why I went to the PostgreSQL IRC channel first...they walked me through generating a co