Strange behavior when printing a 4G+ integer on 32bits platform.
Hello.
I want to represent up to a few hundreds gigabytes for file size.
On 32bits platform, I noticed that
my $value = ...;
printf("%u\n", $value);
prints 4294967295 if $value >= 4294967295 whereas
my $value = ...;
printf("%s\n", $value);
and
use Math::BigInt;
my $value = ...;
printf("%s\n", Math::BigInt->new($value)->bstr());
print correct value even when $value >= 4294967295.
Is it guaranteed (e.g. described as language specification) that
printf("%s\n", $value) will print correct value for any environment?
--
To unsubscribe, e-mail: beginners-unsubscr...@perl.org
For additional commands, e-mail: beginners-h...@perl.org
http://learn.perl.org/
Strange behavior when printing a 4G+ integer on 32bits platform.
Hello.
I want to represent up to a few hundreds gigabytes for file size.
On 32bits platform, I noticed that
my $value = ...;
printf("%u\n", $value);
prints 4294967295 if $value >= 4294967295 whereas
my $value = ...;
printf("%s\n", $value);
and
use Math::BigInt;
my $value = ...;
printf("%s\n", Math::BigInt->new($value)->bstr());
print correct value even when $value >= 4294967295.
Is it guaranteed (e.g. described as language specification) that
printf("%s\n", $value) will print correct value for any environment?
--
To unsubscribe, e-mail: beginners-unsubscr...@perl.org
For additional commands, e-mail: beginners-h...@perl.org
http://learn.perl.org/
Re: Strange behavior when printing a 4G+ integer on 32bits platform.
On 2018/11/15 9:27, sisyphus wrote:
> Are you actually encountering files larger than 1e15 bytes ?
On 32bit kernels, filesystems would not allow such large files.
For example, max size for xfs filesystem is 16TB.
Even on 64bit userspace, nobody will want to upload such large file
via network.
> If not, then either:
>
> printf("%s\n", $sb->size);
> or
> printf("%.15e\n", $sb->size;
> or
> print $sb->size;
>
> should be fine.
I see. Thank you.
--
To unsubscribe, e-mail: beginners-unsubscr...@perl.org
For additional commands, e-mail: beginners-h...@perl.org
http://learn.perl.org/
Re: Strange behavior when printing a 4G+ integer on 32bits platform.
I received a comment from Peter John Acklam (current bigint maintainer).
Forwarded Message
Date: Wed, 14 Nov 2018 20:08:27 +0100
Hello
This has actually nothing to do with Math::BigInt. The issue here is that
the conversion "u", converts a number to an unsigned integer before the
integer is printed as a string. On 32 bit systems, an unsigned integer has
values between 0 and 2^32-1, inclusive. On 64 bit systems, the values are
between 0 and 2^64-1. Using input values outside of this range is not a
good idea. It seems that if the input is larger than the maximum value, it
is truncated to the maximum value, but I wouldn't rely on this. On my 64
bit system, the input value is truncated:
$ perl -we 'printf "%u\n", "1e30"'
18446744073709551615
On the other hand, when the input value is smaller than zero, the input is
not truncated to zero on my system:
$ perl -we 'printf "%u\n", "-1e30"'
9223372036854775808
As for the "s" conversion, it never converts the input to an integer (or
float). Strings are passed right through, unmodified. So that's why I get
$ perl -we 'printf "%s\n", "1e30"'
1e30
If you want "1e30" to be displayed as "100...", Math::BigInt is handy for
converting "1e30" to "100...":
$ perl -MMath::BigInt -we 'printf "%s\n", Math::BigInt->new("1e30")->bstr()'
100
Regarding the question "Is it guaranteed (e.g. described as language
specification) that printf("%s\n", $value) will print correct value
for any environment?"
Yes, the "s" conversion passes strings through unmodified.
If you have more questions or want more clarification, don't hesitate to
ask again.
Best regards,
Peter
On 2018/11/13 23:57, Mike Flannigan wrote:
>
> I don't have an answer for you, but I find this
> interesting. I note the same issue in 64bit
> up near
> 18446744073709551615
>
> I'm guessing the guy who wrote
> Math::BigInt
> may have the answer.
>
>
> Mike
>
>
> On 11/13/2018 8:07 AM, Tetsuo Handa wrote:
>> Hello.
>>
>> I want to represent up to a few hundreds gigabytes for file size.
>>
>> On 32bits platform, I noticed that
>>
>> my $value = ...;
>> printf("%u\n", $value);
>>
>> prints 4294967295 if $value >= 4294967295 whereas
>>
>> my $value = ...;
>> printf("%s\n", $value);
>>
>> and
>>
>> use Math::BigInt;
>> my $value = ...;
>> printf("%s\n", Math::BigInt->new($value)->bstr());
>>
>> print correct value even when $value >= 4294967295.
>>
>> Is it guaranteed (e.g. described as language specification) that
>> printf("%s\n", $value) will print correct value for any environment?
>>
>
--
To unsubscribe, e-mail: beginners-unsubscr...@perl.org
For additional commands, e-mail: beginners-h...@perl.org
http://learn.perl.org/
Re: Strange behavior when printing a 4G+ integer on 32bits platform.
On Thu, Nov 15, 2018 at 1:49 AM Tetsuo Handa <
penguin-ker...@i-love.sakura.ne.jp> wrote:
> 32bits userspace on 64bits kernel:
> # truncate -s 9223372036854775807 test
> # perl -e 'use File::stat; my $sb = lstat("test"); printf("%s\n",
$sb->size);'
> 9.22337203685478e+18
> # perl -e 'use File::stat; my $sb = lstat("test"); printf("%u\n",
$sb->size);'
> 4294967295
Note that "%s" is providing an inaccurate (rounded) result.
Doing :
"%s\n", $sb->size
will provide correct results if the file size is less than or equal
to 1e+15 bytes but, for larger sizes, the value that is output is subject
to rounding.
Doing :
"%.15e\n", $sb->size
provides an improvement on "%s" in that, for files up to
9.007199254740992e+015 bytes, it will provide a correct figure:
If you're encountering files larger than 9.007199254740992e+015 bytes, the
best solution is to simply build perl with -Duse64bitint, though building
with -Duselongdouble would probably also address this issue by providing a
64-bit precision NV.
I do, of course, understand that such solutions might be inapplicable
regarding your situation.
Are you actually encountering files larger than 1e15 bytes ?
If not, then either:
printf("%s\n", $sb->size);
or
printf("%.15e\n", $sb->size;
or
print $sb->size;
should be fine.
Cheers,
Rob
Re: Strange behavior when printing a 4G+ integer on 32bits platform.
On 2018/11/14 21:46, sisyphus wrote:
> On Wed, Nov 14, 2018 at 10:20 PM Tetsuo Handa
> wrote:
>
>> Even on 32bit environments (at least for Linux), lstat() calls 64bit version
>
Here is some test results on Linux.
32bits userspace on 32bits kernel:
# truncate -s 17592186044415 test
# perl -e 'use File::stat; my $sb = lstat("test"); printf("%s\n", $sb->size);'
17592186044415
# perl -e 'use File::stat; my $sb = lstat("test"); printf("%u\n", $sb->size);'
4294967295
32bits userspace on 64bits kernel:
# truncate -s 9223372036854775807 test
# perl -e 'use File::stat; my $sb = lstat("test"); printf("%s\n", $sb->size);'
9.22337203685478e+18
# perl -e 'use File::stat; my $sb = lstat("test"); printf("%u\n", $sb->size);'
4294967295
64bits userspace on 64bits kernel:
# truncate -s 9223372036854775807 test
# perl -e 'use File::stat; my $sb = lstat("test"); printf("%s\n", $sb->size);'
9223372036854775807
# perl -e 'use File::stat; my $sb = lstat("test"); printf("%u\n", $sb->size);'
9223372036854775807
> When I check on Windows, I find that the value is actually an NV (not an IV
> as I had expected).
> I expect it's the same for you, in which case it's a 53-bit double
> (unless $Config{nvsize} reports a value greater than 8).
>
> You can test for this using Devel::Peek:
>
64bits userspace on 64bits kernel:
NV is 8 bytes
SV = IV(0x1f98730) at 0x1f98740
REFCNT = 1
FLAGS = (IOK,pIOK)
IV = 9223372036854775807
32bits userspace on 64bits kernel:
NV is 8 bytes
SV = NV(0x95d7608) at 0x95b45b8
REFCNT = 1
FLAGS = (NOK,pNOK)
NV = 9.22337203685478e+18
Since accurate file size value is lost as soon as it is stored into $sb, doing
my $x = $sb->size;
$value = Math::BigInt->new("$x");
won't help. I will use "%s" without Math::BigInt.
--
To unsubscribe, e-mail: beginners-unsubscr...@perl.org
For additional commands, e-mail: beginners-h...@perl.org
http://learn.perl.org/
Re: Strange behavior when printing a 4G+ integer on 32bits platform.
On Wed, Nov 14, 2018 at 10:20 PM Tetsuo Handa <
penguin-ker...@i-love.sakura.ne.jp> wrote:
> Even on 32bit environments (at least for Linux), lstat() calls 64bit
version
When I check on Windows, I find that the value is actually an NV (not an IV
as I had expected).
I expect it's the same for you, in which case it's a 53-bit double
(unless $Config{nvsize} reports a value greater than 8).
You can test for this using Devel::Peek:
use strict;
use warnings;
use Config;
use Devel::Peek;
die "Usage 'perl try.pl filename'"
unless $ARGV[0];
print "NV is ", $Config{nvsize}, " bytes\n";
my @s = lstat($ARGV[0]);
Dump $s[7];
print "\n\n\n";
Dump $s[8];
__END__
Outputs:
NV is 8 bytes
SV = NV(0x1dbaccc) at 0x1d8e95c
REFCNT = 1
FLAGS = (NOK,pNOK)
NV = 7948
SV = IV(0x1d8e9a8) at 0x1d8e9ac
REFCNT = 1
FLAGS = (IOK,pIOK)
IV = 1481534816
Cheers,
Rob
Re: Strange behavior when printing a 4G+ integer on 32bits platform.
On 2018/11/14 19:59, sisyphus wrote:
> On Wed, Nov 14, 2018 at 9:08 PM Tetsuo Handa
> wrote
>
>> how can I pass $sb->size to Math::BigInt->new() as a string (assuming that
>> $sb->size is an integer) ?
>
> To answer the question, you can do:
>
> my $x = $sb->size;
> $value = Math::BigInt->new("$x");
Wow! I didn't notice that quoting a variable makes a string...
>
> But doing so is likely to be completely pointless.
> I'm thinking that the real problem is "How, on a 32-bit integer build of
> perl, does one get lstat() to return a correct value for 'size' when the
> size of the file is greater than 4294967295 bytes ?"
Even on 32bit environments (at least for Linux), lstat() calls 64bit version,
and thus capable of storing $sb->size > 4GB. Thus, I think it is no problem.
strace perl -e 'use File::stat; my $sb =
lstat("CentOS-5.11-x86_64-bin-DVD-1of2.iso"); print $sb->size;'
lstat64("CentOS-5.11-x86_64-bin-DVD-1of2.iso", {st_mode=S_IFREG|0644,
st_size=4674615296, ...}) = 0
write(1, "4674615296", 10) = 10
Thank you.
--
To unsubscribe, e-mail: beginners-unsubscr...@perl.org
For additional commands, e-mail: beginners-h...@perl.org
http://learn.perl.org/
Re: Strange behavior when printing a 4G+ integer on 32bits platform.
On Wed, Nov 14, 2018 at 9:08 PM Tetsuo Handa <
penguin-ker...@i-love.sakura.ne.jp> wrote
> how can I pass $sb->size to Math::BigInt->new() as a string (assuming
that $sb->size is an integer) ?
To answer the question, you can do:
my $x = $sb->size;
$value = Math::BigInt->new("$x");
But doing so is likely to be completely pointless.
I'm thinking that the real problem is "How, on a 32-bit integer build of
perl, does one get lstat() to return a correct value for 'size' when the
size of the file is greater than 4294967295 bytes ?"
I don't perform such exercises with perl, but I'm thinking that the answer
to that question is "This cannot be done".
Assuming that's correct, the simplest solution in my opinion is to build
your perl with 64-bit integer support - which is something that can be done
even with 32-bit compilers.
Given that the OS knows the actual size of the file, another option would
be to have perl query the OS, rather than calling lstat or stat. (I don't
know the details of how to do that - it would depend to at least some
extent on which OS you're running.)
Cheers,
Rob
Re: Strange behavior when printing a 4G+ integer on 32bits platform.
Hello, sisyphus.
Thank you for your answer.
> That can fail if $value is so big that it requires more than 15 decimal
> digits to express it accurately.
I want to use like
(...snipped...)
my $sb = lstat($file) || next;
next unless (S_ISREG($sb->mode) && $sb->size);
(...snipped...)
$curl->setopt(CURLOPT_URL, sprintf('%s/size=%s/stamp=%s/%s', $cgi_path,
$sb->size, $stamp, $filename));
(...snipped...)
and therefore 15 decimal digits will be enough. Who wants to upload 10TB+ file?
;-)
But for the record who wants to use such large value,
> Note that even with Math::BigInt it's important that you assign the value
> as a string:
>
> C:\_32>perl -MMath::BigInt -le "printf '%s',
> Math::BigInt->new('901234567890123456789');"
> 901234567890123456789
>
> If you assign it as number (unquoted):
>
> C:\_32>perl -MMath::BigInt -le "printf '%s',
> Math::BigInt->new(901234567890123456789);"
> 90123456789012300
how can I pass $sb->size to Math::BigInt->new() as a string (assuming that
$sb->size is an integer) ?
--
To unsubscribe, e-mail: beginners-unsubscr...@perl.org
For additional commands, e-mail: beginners-h...@perl.org
http://learn.perl.org/
Re: Strange behavior when printing a 4G+ integer on 32bits platform.
On 11/13/2018 8:07 AM, Tetsuo Handa wrote:
> Hello.
>
> I want to represent up to a few hundreds gigabytes for file size.
>
> On 32bits platform, I noticed that
>
>my $value = ...;
>printf("%u\n", $value);
>
> prints 4294967295 if $value >= 4294967295 whereas
>
>my $value = ...;
>printf("%s\n", $value);
>
That can fail if $value is so big that it requires more than 15 decimal
digits to express it accurately.
For example:
C:\_32>perl -le "printf '%s', 901234567890123456789;"
9.01234567890123e+020
As you can see, it's printing the value as 90123456789012300
> and
>
>use Math::BigInt;
>my $value = ...;
>printf("%s\n", Math::BigInt->new($value)->bstr());
>
> print correct value even when $value >= 4294967295.
>
> Is it guaranteed (e.g. described as language specification) that
> printf("%s\n", $value) will print correct value for any environment?
>
If you're dealing with integer values that overflow perl's integer type
(IV), then you're generally better off using a module that accommodates
larger integers.
Math::BigInt is fine for this, though it can be rather slow if you're doing
lots of large calculations.
Other options include Math::Int64 (64-bit integer support),
Math::Int128(128-bit integer support), Math::GMP(multiple precision) and
Math::GMPz (multiple precision).
Note that even with Math::BigInt it's important that you assign the value
as a string:
C:\_32>perl -MMath::BigInt -le "printf '%s',
Math::BigInt->new('901234567890123456789');"
901234567890123456789
If you assign it as number (unquoted):
C:\_32>perl -MMath::BigInt -le "printf '%s',
Math::BigInt->new(901234567890123456789);"
90123456789012300
Similarly, assign as a string when using the other alternative modules
that I mentioned.
Note also that you can just print() the value:
C:\_32>perl -MMath::BigInt -le "print
Math::BigInt->new('901234567890123456789');"
901234567890123456789
There's no need to invoke printf(), and no need to call the bstr() method.
Cheers,
Rob
--
To unsubscribe, e-mail: beginners-unsubscr...@perl.org
For additional commands, e-mail: beginners-h...@perl.org
http://learn.perl.org/
Re: Strange behavior when printing a 4G+ integer on 32bits platform.
I don't have an answer for you, but I find this
interesting. I note the same issue in 64bit
up near
18446744073709551615
I'm guessing the guy who wrote
Math::BigInt
may have the answer.
Mike
On 11/13/2018 8:07 AM, Tetsuo Handa wrote:
Hello.
I want to represent up to a few hundreds gigabytes for file size.
On 32bits platform, I noticed that
my $value = ...;
printf("%u\n", $value);
prints 4294967295 if $value >= 4294967295 whereas
my $value = ...;
printf("%s\n", $value);
and
use Math::BigInt;
my $value = ...;
printf("%s\n", Math::BigInt->new($value)->bstr());
print correct value even when $value >= 4294967295.
Is it guaranteed (e.g. described as language specification) that
printf("%s\n", $value) will print correct value for any environment?
--
To unsubscribe, e-mail: beginners-unsubscr...@perl.org
For additional commands, e-mail: beginners-h...@perl.org
http://learn.perl.org/
Strange behavior when printing a 4G+ integer on 32bits platform.
Hello.
I want to represent up to a few hundreds gigabytes for file size.
On 32bits platform, I noticed that
my $value = ...;
printf("%u\n", $value);
prints 4294967295 if $value >= 4294967295 whereas
my $value = ...;
printf("%s\n", $value);
and
use Math::BigInt;
my $value = ...;
printf("%s\n", Math::BigInt->new($value)->bstr());
print correct value even when $value >= 4294967295.
Is it guaranteed (e.g. described as language specification) that
printf("%s\n", $value) will print correct value for any environment?
--
To unsubscribe, e-mail: beginners-unsubscr...@perl.org
For additional commands, e-mail: beginners-h...@perl.org
http://learn.perl.org/
