RE: Perl and memory...
At 12:30 PM 7/28/2009 -0700, Jan Dubois wrote: >It is also not clear to me if you were looking at physical or >virtual memory allocation. In some ways it doesn't make sense >to obsess about returning memory to the OS too much: if you don't >use it anymore, it will just get paged out to disk. And other >processes have their own virtual memory anyways. Sergei's test script showed the physical and virtual usage dropping after thread joining. In my situation (Solaris, ActivePerl) I believe the problem was exceeding some 32 bit resource limit. It was a 100 thread 2GB of memory monster that ran for several days. I had to reduce the thread usage to 60 even after adding the multiprocess component to keep it from crashing with resource messages. The problem was that individual threads would occasionally need very large memory spaces. Once they were done and moved on to smaller tasks, the memory would never be made available to other threads. So once enough threads had run across large memory tasks, the whole process would crash. So when ur obstacle is resource limits, regaining memory is critical. -- REMEMBER THE WORLD TRADE CENTER ---=< WTC 911 >=-- "...ne cede malis" 0100 ___ Perl-Win32-Users mailing list Perl-Win32-Users@listserv.ActiveState.com To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
RE: Perl and memory...
Hi, I am using perl embedded on C + wxWidgets with a threading mechnism. If a thread is never join, I can see clearly the amount of memory growing after each thread run. However, if it is joined, the amount of memory used gets freed. I had checked that on Windows 2000. Kind regards, Jean-Philippe -Original Message- From: perl-win32-users-boun...@listserv.activestate.com [mailto:perl-win32-users-boun...@listserv.activestate.com] On Behalf Of Chris Wagner Sent: Tuesday, July 28, 2009 8:57 PM To: perl-win32-users@listserv.ActiveState.com Subject: Re: Perl and memory... At 06:48 PM 7/28/2009 +0300, Serguei Trouchelle wrote: >I'm not sure. Ending thread on Windows deallocates memory as it said in MSDN, but I'm not exactly sure how Perl handles >all this stuff. So that could be Windows specific? The application I made was on Solaris. I actually thought it was a memory leak in my code. After I found out Perl couldn't free() memory I gave up on trying to shrink the process size and implemented a multiprocess system to deal with the memory issue. It was ActiveState Perl though. Jan? -- REMEMBER THE WORLD TRADE CENTER ---=< WTC 911 >=-- "...ne cede malis" 0100 ___ Perl-Win32-Users mailing list Perl-Win32-Users@listserv.ActiveState.com To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs ___ Perl-Win32-Users mailing list Perl-Win32-Users@listserv.ActiveState.com To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
Re: Perl and memory...
Chris Wagner wrote: >> I'm not sure. Ending thread on Windows deallocates memory as it said in > MSDN, but I'm not exactly sure how Perl handles >> all this stuff. > So that could be Windows specific? The application I made was on Solaris. Yes, it could. From the other hand, once I had a script on Linux, that had to iterate an array of ~1000 big records (100M memory total), and replacing "foreach my $obj (@objects)" with "while (my $obj = shift @objects)" and using weak references (Scalar::Util) in these objects actually lead to memory usage decrease on every iteration. But this method doesn't work on Windows, so, I believe, it's up to OS. > I actually thought it was a memory leak in my code. After I found out Perl > couldn't free() memory I gave up on trying to shrink the process size and > implemented a multiprocess system to deal with the memory issue. Well, you may try the script I sent in previous message to check how threads work on Solaris. If they actually free memory on thread completion, it would be good. -- Serguei Trouchelle ___ Perl-Win32-Users mailing list Perl-Win32-Users@listserv.ActiveState.com To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
RE: Perl and memory...
On Tue, 28 Jul 2009, Chris Wagner wrote: > At 06:48 PM 7/28/2009 +0300, Serguei Trouchelle wrote: > > I'm not sure. Ending thread on Windows deallocates memory as it said > > in MSDN, but I'm not exactly sure how Perl handles all this stuff. > > So that could be Windows specific? The application I made was on > Solaris. I actually thought it was a memory leak in my code. After I > found out Perl couldn't free() memory I gave up on trying to shrink > the process size and implemented a multiprocess system to deal with > the memory issue. It was ActiveState Perl though. Jan? I'm somewhat surprised that even the Windows version returns memory back to the OS. Perl on Windows used to use separate heaps for each interpreter and they got freed when the interpreter got freed. But nowadays everything is using malloc/free from the C runtime library. So I guess it really depends on the runtime library if it will return heap blocks back to the system or not. It is also not clear to me if you were looking at physical or virtual memory allocation. In some ways it doesn't make sense to obsess about returning memory to the OS too much: if you don't use it anymore, it will just get paged out to disk. And other processes have their own virtual memory anyways. Cheers, -Jan ___ Perl-Win32-Users mailing list Perl-Win32-Users@listserv.ActiveState.com To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
Re: Perl and memory...
At 06:48 PM 7/28/2009 +0300, Serguei Trouchelle wrote: >I'm not sure. Ending thread on Windows deallocates memory as it said in MSDN, but I'm not exactly sure how Perl handles >all this stuff. So that could be Windows specific? The application I made was on Solaris. I actually thought it was a memory leak in my code. After I found out Perl couldn't free() memory I gave up on trying to shrink the process size and implemented a multiprocess system to deal with the memory issue. It was ActiveState Perl though. Jan? -- REMEMBER THE WORLD TRADE CENTER ---=< WTC 911 >=-- "...ne cede malis" 0100 ___ Perl-Win32-Users mailing list Perl-Win32-Users@listserv.ActiveState.com To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
Re: Perl and memory...
Chris Wagner wrote:
>> Or, as your question partially suggests, use threads: ending a thread will
> release the memory back to OS.
>
> Really?
Yes, here's an example (takes about 200M of memory and releases it):
_
#!/usr/bin/perl -w
use strict;
use warnings;
use threads;
$| = 1;
sub start_thread {
my $count = shift;
my @var = 1 .. $count;
print 'Ok, we ate some memory...';
<>;
}
my $thr = threads->create('start_thread', 4_000_000);
$thr->join();
print 'We freed it';
<>;
print 'Let us eat again... ';
$thr = threads->create('start_thread', 4_000_000);
$thr->join();
print 'We freed it';
<>;
___
> Is that documented anywhere? Knowing that could've saved me a lot
> of trouble on a massively threaded long running application I made a while
> ago.
I'm not sure. Ending thread on Windows deallocates memory as it said in MSDN,
but I'm not exactly sure how Perl handles
all this stuff.
> Perlthrtut should have that kind of information. Who maintains that?
perl5_porters, I believe.
--
Serguei Trouchelle
___
Perl-Win32-Users mailing list
Perl-Win32-Users@listserv.ActiveState.com
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
Re: Perl and memory...
At 01:31 AM 7/25/2009 +0300, Serguei Trouchelle wrote: >Or, as your question partially suggests, use threads: ending a thread will release the memory back to OS. Really? Is that documented anywhere? Knowing that could've saved me a lot of trouble on a massively threaded long running application I made a while ago. Perlthrtut should have that kind of information. Who maintains that? -- REMEMBER THE WORLD TRADE CENTER ---=< WTC 911 >=-- "...ne cede malis" 0100 ___ Perl-Win32-Users mailing list Perl-Win32-Users@listserv.ActiveState.com To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
Re: Perl and memory...
Amine wrote:
> How can i 'force' Perl to return the used memory ?
You cannot. This memory is actually free, you can still use it in your program,
as you can see when you run "func" once
again.
Or, as your question partially suggests, use threads: ending a thread will
release the memory back to OS.
> When you will run this script , you will see that the
> second call to func() does take a lot of time(much more
> than the first call) , why ?
It takes the same time for me (I've changed one million to five):
#!/usr/bin/perl;
use strict;
use warnings;
use Time::HiRes qw/time/;
my $time = time;
sub func
{
my @b;
for (my $i=0;$i<500;$i++)
{ $b[$i] = 'Perl';}
undef @b;
print '@b memory returned..';
print $time - time, "sec\n";
}
func;
print $time - time, "sec\n";
func;
print $time - time, "sec\n";
func;
print $time - time, "sec\n";
print "end ...\n";
--
Serguei Trouchelle
___
Perl-Win32-Users mailing list
Perl-Win32-Users@listserv.ActiveState.com
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
Re: Perl and memory...
- Original Message -
From: "rocku"
To: "Amine"
Cc:
Sent: Friday, July 24, 2009 3:20 PM
Subject: Re: Perl and memory...
> To answer your first question, take a look at Perl's FAQ:
> http://perldoc.perl.org/perlfaq3.html#How-can-I-free-an-array-or-hash-so-my-program-shrinks?
>
Hello,
They wrote:
"Memory allocated to global variables can be reused (within your program)
by using undef()ing and/or delete()."
I have 128 meg free memory on the system where i am testing , now even
if i use a global variable and i run the script to fill the array it will
take 60 meg bytes,
and when i undef the array 18.5 meg bytes will return to the system,
now how does Perl doesn't return more memory ?
there is still more than 40 meg bytes remaining?
And suppose we are using big arrays we will soon get out of memory, is it
not a problem ?
Regards,
Amine.
> Amine pisze:
Regards,
Amine.
>> Hi all,
>>
>> Look at the following
>> script: ---
>>
>> use Thread qw(:DEFAULT async yield); sub func
>> {
>> my @b;
>> for ($i=0;$i<100;$i++)
>> { $b[$i] = 'Perl';}
>> ;
>> undef @b;
>> print '@b memory returned..';
>> ;
>> }
>>
>> @param = ();
>> my $t = Thread->new(\&func, @param); $result = $t->join;
>>
>> print "End of thread...\n";
>> ;
>>
>> --
>>
>> On Windows click on Ctrl-Alt-del and click on Performance to look at the
>> available memory. When you will run this script and the line print "End
>> of thread...\n"; is executed all the memory used will be returned by
>> Perl.
>>
>> But when the line 'undef @b' is executed , all the used memory is not
>> returned. (please look on the Task manager)...
>> So my question is:
>> How can i 'force' Perl to return the used memory ?
>>
>>
>> My second question is this:
>>
>> Look at the following script:
>>
>> ---
>>
>> sub func
>> {
>> my @b;
>> for ($i=0;$i<100;$i++)
>> { $b[$i] = 'Perl';}
>> ;
>> undef @b;
>> print '@b memory returned..';
>> ;
>> }
>>
>> func;
>> func;
>>
>> print "end ...\n";
>> ;
>>
>>
>>
>>
>> When you will run this script , you will see that the second call to
>> func() does take a lot of time(much more than the first call) , why ?
>>
>>
>>
>> Regards,
>> Amine.
>>
>>
>> ___
>> Perl-Win32-Users mailing list
>> Perl-Win32-Users@listserv.ActiveState.com
>> To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
>>
>
___
Perl-Win32-Users mailing list
Perl-Win32-Users@listserv.ActiveState.com
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
Re: Perl and memory...
To answer your first question, take a look at Perl's FAQ:
http://perldoc.perl.org/perlfaq3.html#How-can-I-free-an-array-or-hash-so-my-program-shrinks?
Amine pisze:
> Hi all,
>
> Look at the following script:
> ---
>
> use Thread qw(:DEFAULT async yield);
>
> sub func
> {
> my @b;
> for ($i=0;$i<100;$i++)
> { $b[$i] = 'Perl';}
> ;
> undef @b;
> print '@b memory returned..';
> ;
> }
>
> @param = ();
> my $t = Thread->new(\&func, @param);
> $result = $t->join;
>
> print "End of thread...\n";
> ;
>
> --
>
> On Windows click on Ctrl-Alt-del and click on Performance to
> look at the available memory.
>
> When you will run this script and the line
> print "End of thread...\n";
> is executed all the memory used will be returned by
> Perl.
>
> But when the line 'undef @b' is executed , all the used memory
> is not returned. (please look on the Task manager)...
> So my question is:
> How can i 'force' Perl to return the used memory ?
>
>
> My second question is this:
>
> Look at the following script:
>
> ---
>
> sub func
> {
> my @b;
> for ($i=0;$i<100;$i++)
> { $b[$i] = 'Perl';}
> ;
> undef @b;
> print '@b memory returned..';
> ;
> }
>
> func;
> func;
>
> print "end ...\n";
> ;
>
>
>
>
> When you will run this script , you will see that the
> second call to func() does take a lot of time(much more
> than the first call) , why ?
>
>
>
> Regards,
> Amine.
>
>
>
>
>
>
>
>
>
>
>
> ___
> Perl-Win32-Users mailing list
> Perl-Win32-Users@listserv.ActiveState.com
> To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
>
___
Perl-Win32-Users mailing list
Perl-Win32-Users@listserv.ActiveState.com
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
Perl and memory...
Hi all,
Look at the following script:
---
use Thread qw(:DEFAULT async yield);
sub func
{
my @b;
for ($i=0;$i<100;$i++)
{ $b[$i] = 'Perl';}
;
undef @b;
print '@b memory returned..';
;
}
@param = ();
my $t = Thread->new(\&func, @param);
$result = $t->join;
print "End of thread...\n";
;
--
On Windows click on Ctrl-Alt-del and click on Performance to
look at the available memory.
When you will run this script and the line
print "End of thread...\n";
is executed all the memory used will be returned by
Perl.
But when the line 'undef @b' is executed , all the used memory
is not returned. (please look on the Task manager)...
So my question is:
How can i 'force' Perl to return the used memory ?
My second question is this:
Look at the following script:
---
sub func
{
my @b;
for ($i=0;$i<100;$i++)
{ $b[$i] = 'Perl';}
;
undef @b;
print '@b memory returned..';
;
}
func;
func;
print "end ...\n";
;
When you will run this script , you will see that the
second call to func() does take a lot of time(much more
than the first call) , why ?
Regards,
Amine.
___
Perl-Win32-Users mailing list
Perl-Win32-Users@listserv.ActiveState.com
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
