64bit linux(ubuntu), so I don't think there's any practical memory limit on a python process in that case I've had python processes up to 8GB personally Marcus
On Fri, Apr 15, 2011 at 10:54 AM, Bill Janssen <[email protected]> wrote: > Marcus <[email protected]> wrote: > > > Bill: > > I'm not sure I follow. > > why would raising the JVM memory to 4GB ever cause a crash in python? > > Our server has 48GB. > > I don't know the specifics of your deployment, but you may not be able > to use that much. 32-bit Python, for instance, won't be able to use it. > Even with 64-bit Python, the OS may place limits on how much memory can > be used by single process. If the Java VM uses too much, the Python VM > will be choked. > > Bill > > > > > thanks > > Marcus > > > > On Fri, Apr 15, 2011 at 7:33 AM, Bill Janssen <[email protected]> wrote: > > > > > Marcus <[email protected]> wrote: > > > > > > > --bcaec53043296dfbfd04a0ece1ac > > > > Content-Type: text/plain; charset=ISO-8859-1 > > > > > > > > we're currently using 4GB max heap. > > > > We recently moved from 2GB to 4GB when we discovered it prevented a > crash > > > > with a certain set of docs. > > > > Marcus > > > > > > I've tried the same workaround with the heap in the past, and I found > it > > > caused NoMemory crashes in the Python side of the house, because the > > > Python VM couldn't get enough memory to operate. So, be careful. > > > > > > > On Thu, Apr 14, 2011 at 5:01 PM, Andi Vajda <[email protected]> > wrote: > > > > > > > > > > > > > > On Thu, 14 Apr 2011, Marcus wrote: > > > > > > > > > > thanks. > > > > >> > > > > >> I have documents that will consistently cause this upon writing > them > > > to > > > > >> the > > > > >> index. let me see if I can reduce them down to the crux of the > crash. > > > > >> granted, these are docs are very large, unruly "bad" data, that > should > > > > >> have > > > > >> never gotten this stage in our pipeline, but I was hoping for a > java > > > or > > > > >> lucene exception. > > > > >> > > > > >> I also get "Java GC overhead" exceptions passed into my code from > time > > > to > > > > >> time, but those manageable, and not crashes. > > > > >> > > > > >> Are there known memory constraint scenarios that force a c++ > > > exception, > > > > >> whereas in a normal Java environment, you would get a memory > error? > > > > >> > > > > > > > > > > Not sure. > > > > > > > > > > > > > > > and just confirming, do "java.lang.OutOfMemoryError" errors pass > into > > > > >> python, or force a crash? > > > > >> > > > > > > > > > > Not sure, I've never seen these as I make sure I've got enough > memory. > > > > > initVM() is the place where you can configure the memory for your > JVM. > > > > > > > > > > Andi.. > > > > > > > > > > > > > > > > > > > >> thanks again > > > > >> Marcus > > > > >> > > > > >> On Thu, Apr 14, 2011 at 2:07 PM, Andi Vajda <[email protected]> > wrote: > > > > >> > > > > >> > > > > >>> On Thu, 14 Apr 2011, Marcus wrote: > > > > >>> > > > > >>> in certain cases when a java/pylucene exception occurs, it gets > > > passed > > > > >>> up > > > > >>> > > > > >>>> in my code, and I'm able to analyze the situation. > > > > >>>> sometimes though, the python process just crashes, and if I > happen > > > to > > > > >>>> be > > > > >>>> in > > > > >>>> top (linux top that is), I see a JCC exception flash up in the > top > > > > >>>> console. > > > > >>>> where can I go to look for this exception, or is it just lost? > > > > >>>> I looked in the locations where a java crash would be located, > but > > > > >>>> didn't > > > > >>>> find anything. > > > > >>>> > > > > >>>> > > > > >>> If you're hitting a crash because of an unhandled C++ exception, > > > running > > > > >>> a > > > > >>> debug build with symbols under gdb will help greatly in tracking > it > > > down. > > > > >>> > > > > >>> An unhandled C++ exception would be a PyLucene/JCC bug. If you > have a > > > > >>> simple way to reproduce this failure, send it to this list. > > > > >>> > > > > >>> Andi.. > > > > >>> > > > > >>> > > > > >> > > > > > > > > --bcaec53043296dfbfd04a0ece1ac-- > > > >
