Hello Noah,
On Mon, Aug 26, 2019 at 11:28:09PM -0700, Noah Misch wrote:
> > > I've run into a problem where floats loose their fractions on the way
> > > from the database to a perl script using DBD::Pg. A minimal reproducer:
> > Am I wrong on this list with my question or was there just too much or
> > too little or too specific information at once? :)
> It's the right list. The issue was complex enough to require some
> uninterrupted study time.
Thanks for getting back to me!
> > > # LANG=C perl ~/t.pl
> > > FOO: 1.1:1.100000 C
> > > 1.1
> > > # LANG=de_DE.UTF-8 perl ~/t.pl
> > > FOO: 1.1:1,000000 de_DE.UTF-8
> > > 1
> > > - so far only macOS's system perl seems to be affected. A perl 5.29.9
> > > compiled myself on the same system does not exhibit the problem and
> > > neither does a Debian testing box.
> I can reproduce it on RHEL 7 with DBD::Pg git head.
That's a relief for sure. :)
> That strtod() call is a
> somewhat-recent addition (added in DBD::Pg 3.6.0, from 2017). What is the
> output of these two commands with each of those Perl installations?
> perl -MDBD::Pg -e 'print $DBD::Pg::VERSION, "\n"'
> LANG=de_DE.UTF-8 perl -MPOSIX -e 'use strict; use warnings;
> setlocale(LC_NUMERIC, ""); print join " ", strtod("1.1"), "\n"'
Here's the output plus some unsolicited version info:
# /usr/bin/perl -V
Summary of my perl5 (revision 5 version 18 subversion 4) configuration:
Platform:
osname=darwin, osvers=18.0, archname=darwin-thread-multi-2level
uname='darwin osx316.apple.com 18.0 darwin kernel version 17.0.0: fri may 4
10:33:38 pdt 2018; root:xnu-4570.1.46.100.2~1development_x86_64 x86_64 '
config_args='-ds -e -Dprefix=/usr -Dccflags=-g -pipe -Dldflags=
-Dman3ext=3pm -Duseithreads -Duseshrplib -Dinc_version_list=none -Dcc=cc'
hint=recommended, useposix=true, d_sigaction=define
useithreads=define, usemultiplicity=define
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=define, use64bitall=define, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags =' -g -pipe -fno-common -DPERL_DARWIN -fno-strict-aliasing
-fstack-protector',
optimize='-Os',
cppflags='-g -pipe -fno-common -DPERL_DARWIN -fno-strict-aliasing
-fstack-protector'
ccversion='', gccversion='4.2.1 Compatible Apple LLVM 10.0.1
(clang-1001.0.37.14)', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -fstack-protector'
libpth=/usr/lib /usr/local/lib
libs=
perllibs=
libc=, so=dylib, useshrplib=true, libperl=libperl.dylib
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' '
cccdlflags=' ', lddlflags=' -bundle -undefined dynamic_lookup
-fstack-protector'
Characteristics of this binary (from libperl):
Compile-time options: HAS_TIMES MULTIPLICITY PERLIO_LAYERS
PERL_DONT_CREATE_GVSV
PERL_HASH_FUNC_ONE_AT_A_TIME_HARD
PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP
PERL_PRESERVE_IVUV PERL_SAWAMPERSAND USE_64_BIT_ALL
USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES
USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE
USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF
USE_REENTRANT_API
Locally applied patches:
/Library/Perl/Updates/<version> comes before system perl directories
installprivlib and installarchlib points to the Updates directory
Built under darwin
Compiled at Apr 1 2019 13:12:58
@INC:
/Library/Perl/5.18/darwin-thread-multi-2level
/Library/Perl/5.18
/Network/Library/Perl/5.18/darwin-thread-multi-2level
/Network/Library/Perl/5.18
/Library/Perl/Updates/5.18.4
/System/Library/Perl/5.18/darwin-thread-multi-2level
/System/Library/Perl/5.18
/System/Library/Perl/Extras/5.18/darwin-thread-multi-2level
/System/Library/Perl/Extras/5.18
.
# /usr/bin/perl -I/tmp/dbdpg/lib/perl5 -MDBD::Pg -e 'print $DBD::Pg::VERSION,
"\n"'
3.7.4
# /usr/bin/perl -MPOSIX -e 'use strict; use warnings; setlocale(LC_NUMERIC,
""); print join " ", strtod("1.1"), "\n"'
1.1 0
# LANG=de_DE.UTF-8 /usr/bin/perl -MPOSIX -e 'use strict; use warnings;
setlocale(LC_NUMERIC, ""); print join " ", strtod("1.1"), "\n"'
1 2
# /tmp/myperl/bin/perl5.29.9 -V
Summary of my perl5 (revision 5 version 29 subversion 9) configuration:
Platform:
osname=darwin
osvers=18.5.0
archname=darwin-thread-multi-2level
uname='darwin mac.fritz.box 18.5.0 darwin kernel version 18.5.0: mon mar 11
20:40:32 pdt 2019; root:xnu-4903.251.3~3release_x86_64 x86_64 '
config_args='-ds -e -Dprefix=/tmp/myperl -Dcc=clang -Dccflags=
-mmacosx-version-min=10.11 -isysroot
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk
-pipe -fno-common -DPERL_DARWIN -fno-strict-aliasing -fstack-protector-strong
-Adefine:cppflags=-no-cpp-precomp -mmacosx-version-min=10.11 -isysroot
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk
-pipe -fno-common -DPERL_DARWIN -fno-strict-aliasing -fstack-protector-strong
-Adefine:ld=clang -Aappend:ldflags=
-Wl,-syslibroot,/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk
-mmacosx-version-min=10.11 -Aappend:lddlflags=
-Wl,-syslibroot,/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk
-mmacosx-version-min=10.11 -Dman3ext=3pm -Duseithreads -Dinc_version_list=none
-Duserelocatableinc -Dusedevel'
hint=recommended
useposix=true
d_sigaction=define
useithreads=define
usemultiplicity=define
use64bitint=define
use64bitall=define
uselongdouble=undef
usemymalloc=n
default_inc_excludes_dot=define
bincompat5005=undef
Compiler:
cc='clang'
ccflags ='-mmacosx-version-min=10.11 -isysroot
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk
-pipe -fno-common -DPERL_DARWIN -fno-strict-aliasing -fstack-protector-strong
-DPERL_USE_SAFE_PUTENV'
optimize='-O3'
cppflags='-no-cpp-precomp -mmacosx-version-min=10.11 -isysroot
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk
-pipe -fno-common -DPERL_DARWIN -fno-strict-aliasing -fstack-protector-strong'
ccversion=''
gccversion='4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.46.3)'
gccosandvers=''
intsize=4
longsize=8
ptrsize=8
doublesize=8
byteorder=12345678
doublekind=3
d_longlong=define
longlongsize=8
d_longdbl=define
longdblsize=16
longdblkind=3
ivtype='long'
ivsize=8
nvtype='double'
nvsize=8
Off_t='off_t'
lseeksize=8
alignbytes=8
prototype=define
Linker and Libraries:
ld='clang'
ldflags =' -mmacosx-version-min=10.14
-Wl,-syslibroot,/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk
-mmacosx-version-min=10.11 -fstack-protector-strong'
libpth=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.1/lib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/lib
/usr/lib
libs=-lpthread -ldbm -ldl -lm -lutil -lc
perllibs=-lpthread -ldl -lm -lutil -lc
libc=
so=dylib
useshrplib=false
libperl=libperl.a
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_dlopen.xs
dlext=bundle
d_dlsymun=undef
ccdlflags=' '
cccdlflags=' '
lddlflags=' -mmacosx-version-min=10.14 -bundle -undefined dynamic_lookup
-Wl,-syslibroot,/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk
-mmacosx-version-min=10.11 -fstack-protector-strong'
Characteristics of this binary (from libperl):
Compile-time options:
HAS_TIMES
MULTIPLICITY
PERLIO_LAYERS
PERL_COPY_ON_WRITE
PERL_DONT_CREATE_GVSV
PERL_IMPLICIT_CONTEXT
PERL_MALLOC_WRAP
PERL_OP_PARENT
PERL_PRESERVE_IVUV
PERL_USE_DEVEL
PERL_USE_SAFE_PUTENV
USE_64_BIT_ALL
USE_64_BIT_INT
USE_ITHREADS
USE_LARGE_FILES
USE_LOCALE
USE_LOCALE_COLLATE
USE_LOCALE_CTYPE
USE_LOCALE_NUMERIC
USE_LOCALE_TIME
USE_PERLIO
USE_PERL_ATOF
USE_REENTRANT_API
USE_THREAD_SAFE_LOCALE
Built under darwin
Compiled at Apr 3 2019 22:40:04
@INC:
/tmp/myperl/lib/site_perl/5.29.9/darwin-thread-multi-2level
/tmp/myperl/lib/site_perl/5.29.9
/tmp/myperl/lib/5.29.9/darwin-thread-multi-2level
/tmp/myperl/lib/5.29.9
# /tmp/myperl/bin/perl5.29.9 -MDBD::Pg -e 'print $DBD::Pg::VERSION, "\n"'
3.7.4
# /tmp/myperl/bin/perl5.29.9 -MPOSIX -e 'use strict; use warnings;
setlocale(LC_NUMERIC, ""); print join " ", strtod("1.1"), "\n"'
1.1 0
# LANG=de_DE.UTF-8 /tmp/myperl/bin/perl5.29.9 -MPOSIX -e 'use strict; use
warnings; setlocale(LC_NUMERIC, ""); print join " ", strtod("1.1"), "\n"'
1 2
While this would seem to suggest that both use a locale-aware strtod in the
POSIX module, the reproducer still holds:
# /usr/bin/perl -I/tmp/dbdpg/lib/perl5 t.pl
1.1
# LANG=de_DE.UTF-8 /usr/bin/perl -I/tmp/dbdpg/lib/perl5 t.pl
1
# /tmp/myperl/bin/perl5.29.9 t.pl
1.1
# LANG=de_DE.UTF-8 /tmp/myperl/bin/perl5.29.9 t.pl
1.1
That would seem to suggest that DBD::Pg in my self-compiled perl uses another
strtod than the POSIX module. The same seems to be the case on Debian testing:
# perl -MDBD::Pg -e 'print $DBD::Pg::VERSION, "\n"'
3.9.1
# perl -MPOSIX -e 'use strict; use warnings; setlocale(LC_NUMERIC, ""); print
join " ", strtod("1.1"), "\n"'
1.1 0
# LANG=de_DE.UTF-8 perl -MPOSIX -e 'use strict; use warnings;
setlocale(LC_NUMERIC, ""); print join " ", strtod("1.1"), "\n"'
1 2
# perl t.pl
1.1
# LANG=de_DE.UTF-8 perl t.pl
1.1
# perl -V
Summary of my perl5 (revision 5 version 28 subversion 1) configuration:
Platform:
osname=linux
osvers=4.9.0
archname=x86_64-linux-gnu-thread-multi
uname='linux localhost 4.9.0 #1 smp debian 4.9.0 x86_64 gnulinux '
config_args='-Dusethreads -Duselargefiles -Dcc=x86_64-linux-gnu-gcc
-Dcpp=x86_64-linux-gnu-cpp -Dld=x86_64-linux-gnu-gcc -Dccflags=-DDEBIAN
-Wdate-time -D_FORTIFY_SOURCE=2 -g -O2
-fdebug-prefix-map=/build/perl-5WfRyb/perl-5.28.1=. -fstack-protector-strong
-Wformat -Werror=format-security -Dldflags= -Wl,-z,relro -Dlddlflags=-shared
-Wl,-z,relro -Dcccdlflags=-fPIC -Darchname=x86_64-linux-gnu -Dprefix=/usr
-Dprivlib=/usr/share/perl/5.28 -Darchlib=/usr/lib/x86_64-linux-gnu/perl/5.28
-Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5
-Dvendorarch=/usr/lib/x86_64-linux-gnu/perl5/5.28 -Dsiteprefix=/usr/local
-Dsitelib=/usr/local/share/perl/5.28.1
-Dsitearch=/usr/local/lib/x86_64-linux-gnu/perl/5.28.1
-Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3
-Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3
-Duse64bitint -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs
-Ud_csh -Ud_ualarm -Uusesfio -Uusenm -Ui_libutil -Ui_xlocale -Uversiononly
-DDEBUGGING=-g -Doptimize=-O2 -dEs -Duseshrplib -Dlibperl=libperl.so.5.28.1'
hint=recommended
useposix=true
d_sigaction=define
useithreads=define
usemultiplicity=define
use64bitint=define
use64bitall=define
uselongdouble=undef
usemymalloc=n
default_inc_excludes_dot=define
bincompat5005=undef
Compiler:
cc='x86_64-linux-gnu-gcc'
ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing
-pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'
optimize='-O2 -g'
cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing
-pipe -I/usr/local/include'
ccversion=''
gccversion='8.3.0'
gccosandvers=''
intsize=4
longsize=8
ptrsize=8
doublesize=8
byteorder=12345678
doublekind=3
d_longlong=define
longlongsize=8
d_longdbl=define
longdblsize=16
longdblkind=3
ivtype='long'
ivsize=8
nvtype='double'
nvsize=8
Off_t='off_t'
lseeksize=8
alignbytes=8
prototype=define
Linker and Libraries:
ld='x86_64-linux-gnu-gcc'
ldflags =' -fstack-protector-strong -L/usr/local/lib'
libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/8/include-fixed
/usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib
/usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
perllibs=-ldl -lm -lpthread -lc -lcrypt
libc=libc-2.28.so
so=so
useshrplib=true
libperl=libperl.so.5.28
gnulibc_version='2.28'
Dynamic Linking:
dlsrc=dl_dlopen.xs
dlext=so
d_dlsymun=undef
ccdlflags='-Wl,-E'
cccdlflags='-fPIC'
lddlflags='-shared -L/usr/local/lib -fstack-protector-strong'
Characteristics of this binary (from libperl):
Compile-time options:
HAS_TIMES
MULTIPLICITY
PERLIO_LAYERS
PERL_COPY_ON_WRITE
PERL_DONT_CREATE_GVSV
PERL_IMPLICIT_CONTEXT
PERL_MALLOC_WRAP
PERL_OP_PARENT
PERL_PRESERVE_IVUV
USE_64_BIT_ALL
USE_64_BIT_INT
USE_ITHREADS
USE_LARGE_FILES
USE_LOCALE
USE_LOCALE_COLLATE
USE_LOCALE_CTYPE
USE_LOCALE_NUMERIC
USE_LOCALE_TIME
USE_PERLIO
USE_PERL_ATOF
USE_REENTRANT_API
Locally applied patches:
[...]
Built under linux
Compiled at Mar 31 2019 11:51:22
@INC:
/etc/perl
/usr/local/lib/x86_64-linux-gnu/perl/5.28.1
/usr/local/share/perl/5.28.1
/usr/lib/x86_64-linux-gnu/perl5/5.28
/usr/share/perl5
/usr/lib/x86_64-linux-gnu/perl/5.28
/usr/share/perl/5.28
/usr/local/lib/site_perl
/usr/lib/x86_64-linux-gnu/perl-base
> > > The text representation of values is whatever strings are
> > > produced and accepted by the input/output conversion functions
> > > for the particular data type.
> PostgreSQL always prints ".", not a locale-specific radix character. A
> locale-ignorant strtod() would suffice in DBD::Pg.
Awesome. Is there anyway I can help with this?
--
Thanks,
Michael
