OK, found some time tonight to play with this. Below is my code, which
introduces the function "convert_array_to_piddle". That function allocates
memory and uses C to perform that actual conversion. The results on my
machine between the normal PDL constructor and the hand-crafted one are
always exactly zero. In this light, I'm now quite curious how you produce
your problem of deep recursion. That message only comes up once in the
core, and I cannot seem to trip it with the piddle-from-array code shown
below.
Note that the way to get PDL to convert undef to the BAD value is to assign
the data type's bad value to that globa, and to turn on the bad flag for
the resulting piddle. This is demonstrated in the section "Convert it to a
piddle normally"
Hope this helps, and let me know if you still hit the recursion guard in
your code using "convert_array_to_piddle".
David
----%<----
use strict;
use warnings;
use PDL;
use Inline 'C';
# Generate a big array.
my $nelem = 20;
my @to_convert;
$#to_convert = $nelem - 1;
# With probability 1 in 100, use the undefined value
my $expected_n_bad_values = 0;
for my $i (0 .. $#to_convert) {
if (rand() > 0.2) {
$to_convert[$i] = rand();
}
else {
$expected_n_bad_values++;
$to_convert[$i] = undef;
}
}
# Convert it to a piddle normally
$PDL::undefval = double->badvalue;
my $normal_pdl = pdl(\@to_convert);
$normal_pdl->badflag(1);
# Convert it to a piddle using the special function
my $fancy_pdl = convert_array_to_piddle(\@to_convert);
if (any $normal_pdl != $fancy_pdl) {
print "Hmm, they disagree\n";
}
else {
print "Full agreement\n";
}
print "normal: $normal_pdl\nfancy : $fancy_pdl\n";
####### Heaving lifting below #######
sub convert_array_to_piddle {
my $array = shift;
my $nelem = scalar(@$array);
# Some way to do this with PDL::new_from_specification? I can't get it
# to work for some reason. Resorting to zeroes.
my $pdl = zeroes(double, $nelem);
# Set the badflag on
$pdl->badflag(1);
# Call the C function to set the pdl buffer to values from the array
set_pdl_double_by_array($array, ${$pdl->get_dataref}, $pdl->badvalue);
return $pdl;
}
__DATA__
__C__
void set_pdl_double_by_array(AV * in_av, SV * pdl_datasv, double badval) {
/* Pull out the data array and cast to doubles */
double * pdl_data = (double*)SvPV_nolen(pdl_datasv);
/* Run through every element of the array */
int i;
for (i = 0; i <= av_len(in_av); i++) {
/* Get this element */
SV ** val_svp = av_fetch(in_av, i, 0);
/* Possible to get zero, though I don't know why we would */
if (val_svp == NULL) croak("Weird: got a null sv from av_fetch??");
/* Good to go, deref for clarity. */
SV * val_sv = *val_svp;
/* Is it undef? */
if (!SvOK(val_sv)) {
/* Yes, set it to bad */
pdl_data[i] = badval;
}
else {
/* No, then get the floating point value */
pdl_data[i] = SvNV(val_sv);
}
}
}
On Thu, Dec 20, 2012 at 10:16 PM, David Mertens <[email protected]>wrote:
> Well, $PDL::undefval is the proper approach. It's odd that you're getting
> this problem still. Let me try to hack something together that doesn't use
> the pdl method itself and see if you still have your recursion limit
> problem. Do you know if you have Inline::C on your machine? That would help
> simplify things.
>
> David
> On Dec 20, 2012 10:05 PM, "Erich Greene" <[email protected]> wrote:
>
>> On Thu, 20 Dec 2012, Craig DeForest wrote:
>>
>> You can specify $PDL::undefval - it will use that instead of 0 for
>>> undefs.
>>>
>>
>> I'd figured out you can't set undefval to anything resembling 'BAD'
>> directly (strings get numified to 0), but that did give me an idea:
>>
>> $PDL::undefval = 'nan';
>> ...;
>> my $pdl = pdl($ref)->inplace->**setnantobad;
>>
>> Worked in test, same error in production, though with a wrinkle: Now the
>> recursion limit is exceeded when I'm using a piddle (e.g., $pdl->avg,
>> $pdl->stdv, $pdl1->corr($pdl2)) rather than when I'm making it.
>>
>> On Thu, 20 Dec 2012, David Mertens wrote:
>>
>>>
>>> I do not know much about the constructor internals, you will probably
>>> need to
>>> wait for Craig or Chris to weigh in. But if there is no way to do it, I
>>> know
>>> exactly how to write an efficient implementation, and could crank
>>> something out
>>> quickly, tomorrow morning if you need it, or tonight if you're in a real
>>> pickle.
>>>
>>
>> It's not a rush, more something I was hoping to have done before the
>> holidays, but I'm interested in whatever you come up with.
>>
>> Thanks.
>>
>> -- Erich
>>
>> On Thu, 20 Dec 2012, Erich Greene wrote:
>>
>>>
>>> Hi,
>>>
>>> I'm fairly new to PDL and this seems like an obvious thing to ask about,
>>> but I
>>> haven't found it addressed in any docs or FAQs or in this list's
>>> archives.
>>>
>>> I'm trying to create piddles from arrays that might include undefined
>>> values
>>> and treat those values as bad in further processing. But when pdl()
>>> encounters
>>> an input value of undef, it puts a 0 into the piddle. Setting 0 as the
>>> bad
>>> value later doesn't work for me because 0 is also a legitimate data
>>> value, and
>>> once the piddle is made, there's no distinguishing an undef treated as 0
>>> from a
>>> real 0. (In fact, any float could be legitimate -- that's why I've been
>>> using
>>> undef for missing data -- so experimenting with badvalue was also a dead
>>> end.)
>>>
>>> Eventually, I came up with this:
>>>
>>> use 5.014;
>>> use warnings;
>>> use PDL::LiteF;
>>>
>>> # make a piddle where undef becomes BAD
>>> sub pdlify {
>>> my @in = map {ref eq 'ARRAY' ? @$_ : $_} @_;
>>> my @out = map {defined($_) ? pdl($_) : pdl(0)->setbadat(0)} @in;
>>> return cat(@out);
>>> }
>>>
>>> This works fine in small-scale tests (my $x = pdlify(1..5,undef,7..10);
>>> say $x;
>>> spits out [1 2 3 4 5 BAD 7 8 9 10] as it should), but in production, one
>>> of two
>>> things eventually happens:
>>>
>>> 1) segfault
>>>
>>> 2) cat: unknown error from the internals:
>>> PDL: Problem with assignment: PDL:Internal Error: data structure
>>> recursion
>>> limit exceeded (max 1000 levels)
>>> This could mean that you have found an infinite-recursion error
>>> in PDL,
>>> or
>>> that you are building data structures with very long dataflow
>>> dependency
>>> chains. You may want to try using sever() to break the
>>> dependency.
>>>
>>> Each call to pdlify is acting on a fresh list of scalars (all numbers or
>>> undef), so there shouldn't be any dependencies to speak of at this point.
>>>
>>> Bottom line, is there an idiom or package already out there that will
>>> take a
>>> list and create a piddle where undefined input values are represented as
>>> BAD
>>> rather than 0, or is there at least some way to do it that doesn't go
>>> down some
>>> mysterious recursive rabbit hole?
>>>
>>> Thanks.
>>>
>>> -- Erich
>>>
>>>
>> ______________________________**_________________
>> Perldl mailing list
>> [email protected]
>> http://mailman.jach.hawaii.**edu/mailman/listinfo/perldl<http://mailman.jach.hawaii.edu/mailman/listinfo/perldl>
>>
>
--
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." -- Brian Kernighan
_______________________________________________
Perldl mailing list
[email protected]
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
