Re: Huge DataNode Virtual Memory Usage

2009-05-12 Thread Steve Loughran
Stefan Will wrote: Raghu, I don't actually have exact numbers from jmap, although I do remember that jmap -histo reported something less than 256MB for this process (before I restarted it). I just looked at another DFS process that is currently running and has a VM size of 1.5GB (~600 resident)

Re: Huge DataNode Virtual Memory Usage

2009-05-11 Thread Stefan Will
-- Stefan > From: Raghu Angadi > Reply-To: > Date: Sun, 10 May 2009 20:06:13 -0700 > To: > Subject: Re: Huge DataNode Virtual Memory Usage > > what do 'jmap' and 'jmap -histo:live' show?. > > Raghu. > > Stefan Will wrote: >> Chris, >>

Re: Huge DataNode Virtual Memory Usage

2009-05-10 Thread Raghu Angadi
.0-b15, mixed mode) Do you know of a specific bug # in the JDK bug database that addresses this ? Cheers, Stefan From: Chris Collins Reply-To: Date: Fri, 8 May 2009 20:34:21 -0700 To: "core-user@hadoop.apache.org" Subject: Re: Huge DataNode Virtual Memory Usage Stefan, there was

Re: Huge DataNode Virtual Memory Usage

2009-05-09 Thread Chris Collins
he.org" Subject: Re: Huge DataNode Virtual Memory Usage Stefan, there was a nasty memory leak in in 1.6.x before 1.6 10. It manifested itself during major GC. We saw this on linux and solaris and dramatically improved with an upgrade. C On May 8, 2009, at 6:12 PM, Stefan Will wrote: Hi,

Re: Huge DataNode Virtual Memory Usage

2009-05-09 Thread Stefan Will
this ? Cheers, Stefan > From: Chris Collins > Reply-To: > Date: Fri, 8 May 2009 20:34:21 -0700 > To: "core-user@hadoop.apache.org" > Subject: Re: Huge DataNode Virtual Memory Usage > > Stefan, there was a nasty memory leak in in 1.6.x before 1.6 10. It > m

Re: Huge DataNode Virtual Memory Usage

2009-05-08 Thread Chris Collins
Stefan, there was a nasty memory leak in in 1.6.x before 1.6 10. It manifested itself during major GC. We saw this on linux and solaris and dramatically improved with an upgrade. C On May 8, 2009, at 6:12 PM, Stefan Will wrote: Hi, I just ran into something rather scary: One of my datano

Huge DataNode Virtual Memory Usage

2009-05-08 Thread Stefan Will
Hi, I just ran into something rather scary: One of my datanode processes that I¹m running with ­Xmx256M, and a maximum number of Xceiver threads of 4095 had a virtual memory size of over 7GB (!). I know that the VM size on Linux isn¹t necessarily equal to the actual memory used, but I wouldn¹t exp