Thanks Chris--The output is attached.
As auxiliary question related to working with lots of data that the program
can't hold all it once: If I have around 100,000 individual files containing
a single column, all of the same length, for I/O speed is it better to
convert all of the files to load using PDL::IO::FastRaw, or is it better to
read it in from .txt each time using rcols()? I've notice that pdls loaded
using rcols aren't taking up any memory (state VC in help vars).
Looking at Craig's output, it could be that I'm not calling $PDL::BIGPDL
properly. When working in a script, is
$c=$PDL::BIGPDL=$a==$b->dummy
the way I allow $c to be >1Gb?
Many Thanks.
Steve
On Sun, Sep 14, 2008 at 2:40 PM, Chris Marshall <[EMAIL PROTECTED]> wrote:
> Could you please run 'perldl -V' and send the output.
> I could not reproduce the problem as I had out of memory
> errors at that point.
>
> Maybe someone with a large memory 64bit OS and
> PDL could take a look further to debug the internals.
>
> As a work around, just loop over your large $a values
> by blocks small enough to run using slicing to extract
> the values of interest for each step.
>
> --Chris
>
> Steve Cicala wrote:
>
>> Hi--I am running perldl on a unix server with 32Gb of memory, but am
>> having
>> a problem with the following operation:Here's a small version that works:
>> perldl> $a=sequence(5)
>>
>> perldl> $b=pdl[1,4,0]
>>
>> perldl> p $c=$a==$b->dummy
>>
>> [
>> [0 1 0 0 0]
>> [0 0 0 0 1]
>> [1 0 0 0 0]
>> ]
>> (I am using this to collect indicies in a that correspond with elements in
>> b:
>> $d=which(maximum (($a==$b->dummy)->xchg(0,1))!=0)
>> ).
>>
>> Now, when I try to use this operation for large numbers (i.e. $a is
>> 480000x1
>> and $b is 500x1) I get:
>>
>> 'multielement piddle in a conditional expression'
>>
>> --And I get this error whether calculating $c on its own, or just sticking
>> the expression that generates $c into the one that generates $d.
>>
>> I have also tried:
>>
>> $c=$PDL::BIGPDL=$a==$b->dummy
>>
>> and get the same error.
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Perldl mailing list
>> [email protected]
>> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
>> ------------------------------------------------------------------------
>>
>> No virus found in this incoming message.
>> Checked by AVG. Version: 7.5.526 / Virus Database: 270.6.21/1671 - Release
>> Date: 9/14/2008 7:16 AM
>>
>>
>
>
perlDL shell v1.33
PDL comes with ABSOLUTELY NO WARRANTY. For details, see the file
'COPYING' in the PDL distribution. This is free software and you
are welcome to redistribute it under certain conditions, see
the same file for details.
Summary of my PDL configuration
VERSION: PDL v2.4.3
$%PDL::Config = {
'BADVAL_PER_PDL' => '0',
'WITH_PROJ' => undef,
'FFTW_TYPE' => 'double',
'FFTW_LIBS' => [
'/lib',
'/usr/lib',
'/usr/local/lib'
],
'WITH_FFTW' => undef,
'GSL_LIBS' => undef,
'WITH_IO_BROWSER' => '0',
'PROJ_INC' => undef,
'WHERE_PLPLOT_INCLUDE' => undef,
'WITH_KARMA' => '0',
'WHERE_KARMA' => undef,
'HTML_DOCS' => '1',
'WHERE_PLPLOT_LIBS' => undef,
'WITH_3D' => '0',
'FFTW_INC' => [
'/usr/include/',
'/usr/local/include'
],
'WITH_POSIX_THREADS' => '1',
'HIDE_TRYLINK' => '1',
'WITH_HDF' => undef,
'HDF_INC' => undef,
'WITH_BADVAL' => '0',
'WITH_GD' => undef,
'FITS_LEGACY' => '1',
'WITH_SLATEC' => undef,
'BADVAL_USENAN' => '0',
'TEMPDIR' => '/tmp',
'PROJ_LIBS' => undef,
'GD_LIBS' => undef,
'GSL_INC' => undef,
'GD_INC' => undef,
'OPTIMIZE' => undef,
'WITH_GSL' => undef,
'HDF_LIBS' => undef,
'MALLOCDBG' => {},
'WITH_PLPLOT' => '0'
};
Summary of my perl5 (revision 5 version 8 subversion 8) configuration:
Platform:
osname=linux, osvers=2.6.18-53.1.14.el5xen,
archname=x86_64-linux-thread-multi
uname='linux xenbuilder2.fedora.redhat.com 2.6.18-53.1.14.el5xen #1 smp tue
feb 19 07:33:17 est 2008 x86_64 x86_64 x86_64 gnulinux '
config_args='-des -Doptimize=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic
-Dversion=5.8.8 -Dmyhostname=localhost [EMAIL PROTECTED] -Dcc=gcc -Dcf_by=Red
Hat, Inc. -Dinstallprefix=/usr -Dprefix=/usr -Dlibpth=/usr/local/lib64 /lib64
/usr/lib64 -Dprivlib=/usr/lib/perl5/5.8.8
-Dsitelib=/usr/lib/perl5/site_perl/5.8.8
-Dvendorlib=/usr/lib/perl5/vendor_perl/5.8.8
-Darchlib=/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi
-Dsitearch=/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi
-Dvendorarch=/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi
-Darchname=x86_64-linux -Dvendorprefix=/usr -Dsiteprefix=/usr -Duseshrplib
-Dusethreads -Duseithreads -Duselargefiles -Dd_dosuid -Dd_semctl_semun -Di_db
-Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio
-Dinstallusrbinperl=n -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr
-Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto
-Ud_endprotoent_r_proto -Ud_setprotoent_r_proto -Ud_endservent_r_proto
-Ud_setservent_r_proto -Dinc_version_list=5.8.7 5.8.6 5.8.5
-Dscriptdir=/usr/bin'
hint=recommended, useposix=true, d_sigaction=define
usethreads=define use5005threads=undef useithreads=define
usemultiplicity=define
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=define use64bitall=define uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe
-Wdeclaration-after-statement -I/usr/local/include -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm',
optimize='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic',
cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe
-Wdeclaration-after-statement -I/usr/local/include -I/usr/include/gdbm'
ccversion='', gccversion='4.1.2 20070925 (Red Hat 4.1.2-27)',
gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='gcc', ldflags =''
libpth=/usr/local/lib64 /lib64 /usr/lib64
libs=-lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc
perllibs=-lresolv -lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
libc=, so=so, useshrplib=true, libperl=libperl.so
gnulibc_version='2.6'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E
-Wl,-rpath,/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE'
cccdlflags='-fPIC', lddlflags='-shared -O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m64 -mtune=generic'
_______________________________________________
Perldl mailing list
[email protected]
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
