Re: [HACKERS] snprintf causes regression tests to fail
The big question is why our own vsnprintf() is not being called from snprintf() in our port file. I have seen this problem before, well, it isn't really a problem I guess. I'm not sure of the gcc compiler options, but On the Microsoft compiler if you specify the option /Gy it separates the functions within the object file, that way you don't load all the functions from the object if they are not needed. If, however, you create a function with the same name as another function, and one is declared in an object compiled with the /Gy option, and the other's object file is not, then if you also use a different function or reference variable in the object file compiled without the /Gy option, then the conflicting function will probably be used. Make sense? I would suggest using macro to redefine snprintf and vnsprintf to avoid the issue: #define snprintf pg_snprintf #define vnsprintf pg_vnsprintf ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] snprintf causes regression tests to fail
I have some new information. If I add the attached patch to snprintf.c, I should see see snprintf() being called printing 0, vsnprintf() printing 1 and dopr(), 2. However, I only see 0 printing in the server logs. I think this means it is finding our /port/snprintf(), but when it calls vsnprintf, it must be using some other version, probably the operating system version that doesn't support %lld. I am also attaching the 'nm' output from libpgport_srv.a which does show vsnprintf as being defined. Win32 doesn't like multiply defined symbols so we use -Wl,--allow-multiple-definition to allow multiple symbols. I bet if I define LONG_LONG_INT_FORMAT as '%I64d' it would pass the regression tests. (I will test now.) Our config/c-library.m4 file confirms that format for MinGW: # MinGW uses '%I64d', though gcc throws an warning with -Wall, The big question is why our own vsnprintf() is not being called from snprintf() in our port file. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 *** snprintf.c Tue Mar 1 19:02:13 2005 --- /laptop/tmp/snprintf.c Tue Mar 1 20:27:21 2005 *** *** 96,101 --- 96,102 int len; va_list args; + puts(0); va_start(args, fmt); len = vsnprintf(str, count, fmt, args); va_end(args); *** *** 109,114 --- 110,116 char *end; str[0] = '\0'; end = str + count - 1; + puts(1); dopr(str, fmt, args, end); if (count 0) end[0] = '\0'; *** *** 178,183 --- 180,186 int realpos; } fmtpar[NL_ARGMAX+1], *fmtparptr[NL_ARGMAX+1]; + puts(2); format_save = format; output = buffer; crypt.o: b .bss d .data t .text 0060 b _a64toi 2ce0 b _CF6464 03c0 t _CIFP 34e0 b _constdatablock 0441 T _crypt 34f0 b _cryptresult 0760 t _des_cipher d _des_ready 06c2 t _des_setkey 00c0 t _ExpandTr 18e0 b _IE3264 0a84 t _init_des 0f0f t _init_perm 0080 t _IP 0400 t _itoa64 3510 b _KS 03a0 t _P32Tr 0100 t _PC1 00e0 b _PC1ROT 0160 t _PC2 08e0 b _PC2ROT b _perm.0 t _permute 0138 t _Rotates 01a0 t _S 1ce0 b _SPE 0040 b _tmp32.1 fseeko.o: b .bss d .data t .text getrusage.o: b .bss d .data t .text U ___udivdi3 U ___umoddi3 U __dosmaperr U __errno U [EMAIL PROTECTED] U [EMAIL PROTECTED] U [EMAIL PROTECTED] T _getrusage inet_aton.o: b .bss d .data t .text U __impmb_cur_max U __imp___pctype U __isctype U [EMAIL PROTECTED] T _inet_aton random.o: b .bss d .data t .text U _lrand48 T _random srandom.o: b .bss d .data t .text U _srand48 T _srandom unsetenv.o: b .bss d .data t .text U _getenv U _malloc U _putenv U _sprintf 0004 T _unsetenv getaddrinfo_srv.o: b .bss d .data t .text U _atoi U _free U [EMAIL PROTECTED] U [EMAIL PROTECTED] U [EMAIL PROTECTED] U _inet_aton U [EMAIL PROTECTED] U _malloc U _memcpy U [EMAIL PROTECTED] 025a T _pg_freeaddrinfo 02ca T _pg_gai_strerror T _pg_getaddrinfo 02f6 T _pg_getnameinfo U _snprintf U [EMAIL PROTECTED] copydir.o: b .bss d .data t .text t ___func__.0 U _AllocateDir 00a5 T _copydir U [EMAIL PROTECTED] U _errcode_for_file_access U _errfinish U _errmsg U _errstart U _FreeDir U _mkdir U _readdir U _snprintf gettimeofday.o: b .bss d .data t .text U ___udivdi3 t _epoch U [EMAIL PROTECTED] 0008 T _gettimeofday U [EMAIL PROTECTED] kill.o: b .bss d .data t .text U __errno U [EMAIL PROTECTED] U [EMAIL PROTECTED] 0015 T _pgkill U _wsprintfA open.o: b .bss d .data t .text U __assert U __errno U __open_osfhandle U __setmode U [EMAIL PROTECTED] U [EMAIL PROTECTED] U [EMAIL PROTECTED] t _openFlagsToCreateFileFlags 0160 T _win32_open rand.o: b .bss d .data t .text t __dorand48 000c D
Re: [HACKERS] snprintf causes regression tests to fail
Bruce Momjian pgman@candle.pha.pa.us writes: I think this means it is finding our /port/snprintf(), but when it calls vsnprintf, it must be using some other version, probably the operating system version that doesn't support %lld. Ya know, I was wondering about that but dismissed it because the routines were all declared in the same file. Windows' linker must behave very oddly to do this. Does it help if you flip the order of the snprintf and vsnprintf functions in snprintf.c? regards, tom lane ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] snprintf causes regression tests to fail
Tom Lane wrote: Bruce Momjian pgman@candle.pha.pa.us writes: I think this means it is finding our /port/snprintf(), but when it calls vsnprintf, it must be using some other version, probably the operating system version that doesn't support %lld. I can confirm that using %I64d for the printf format allows the regression tests to pass for int8. Ya know, I was wondering about that but dismissed it because the routines were all declared in the same file. Windows' linker must behave very oddly to do this. Yep, quite unusual. Does it help if you flip the order of the snprintf and vsnprintf functions in snprintf.c? Testing now. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] snprintf causes regression tests to fail
Tom Lane wrote: Bruce Momjian pgman@candle.pha.pa.us writes: I think this means it is finding our /port/snprintf(), but when it calls vsnprintf, it must be using some other version, probably the operating system version that doesn't support %lld. Ya know, I was wondering about that but dismissed it because the routines were all declared in the same file. Windows' linker must behave very oddly to do this. Does it help if you flip the order of the snprintf and vsnprintf functions in snprintf.c? Yes, it fixes the problem and I have applied the reordering with a comment. I will start working on fixing the large fmtpar allocations now. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] snprintf causes regression tests to fail
Bruce Momjian pgman@candle.pha.pa.us writes: Tom Lane wrote: Does it help if you flip the order of the snprintf and vsnprintf functions in snprintf.c? Yes, it fixes the problem and I have applied the reordering with a comment. Fascinating. I will start working on fixing the large fmtpar allocations now. Quite honestly, this code is not worth micro-optimizing because it is fundamentally broken. See my other comments in this thread. Can we find a BSD-license implementation that follows the spec? regards, tom lane ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] snprintf causes regression tests to fail
Tom Lane wrote: Bruce Momjian pgman@candle.pha.pa.us writes: Tom Lane wrote: Does it help if you flip the order of the snprintf and vsnprintf functions in snprintf.c? Yes, it fixes the problem and I have applied the reordering with a comment. Fascinating. I will start working on fixing the large fmtpar allocations now. Quite honestly, this code is not worth micro-optimizing because it is fundamentally broken. See my other comments in this thread. I am working on something that just counts the '%' characters in the format string and allocates an array that size. Can we find a BSD-license implementation that follows the spec? I would think NetBSD would be our best bet. I will download it and take a look. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] snprintf causes regression tests to fail
Bruce Momjian wrote: Tom Lane wrote: Bruce Momjian pgman@candle.pha.pa.us writes: Tom Lane wrote: Does it help if you flip the order of the snprintf and vsnprintf functions in snprintf.c? Yes, it fixes the problem and I have applied the reordering with a comment. Fascinating. I will start working on fixing the large fmtpar allocations now. Quite honestly, this code is not worth micro-optimizing because it is fundamentally broken. See my other comments in this thread. I am working on something that just counts the '%' characters in the format string and allocates an array that size. Can we find a BSD-license implementation that follows the spec? I would think NetBSD would be our best bet. I will download it and take a look. Oops, I remember now that NetBSD doesn't support it. I think FreeBSD does. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] snprintf causes regression tests to fail
Nicolai Tufar wrote: Linux and Solaris 10 x86 pass regression tests fine when I force the use of new snprintf(). The problem should be win32 - specific. I will investigate it throughly tonight. Can someone experienced in win32 what can possibly be the problem? Yea, I am confused too because my BSD uses the new snprintf.c code was well. Magnus, what failures are you seeing on Win32? -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] snprintf causes regression tests to fail
Linux and Solaris 10 x86 pass regression tests fine when I force the use of new snprintf(). The problem should be win32 - specific. I will investigate it throughly tonight. Can someone experienced in win32 what can possibly be the problem? Nick On Sun, 27 Feb 2005 19:07:16 +0100, Magnus Hagander [EMAIL PROTECTED] wrote: Hi! The new snpritnf code appears not to deal with 64-bit ints. I'm getting failures on win32 for int8 as well as all the time related tests (win32 uses int8 for tinmestamps). Removing the snprintf code and falling back to the OS code makes everything pass again. I would guess this affects int8 etc on other platforms as well (assuming they use our snprintf and not the libc one), but I haven't checked it. //Magnus ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] snprintf causes regression tests to fail
Linux and Solaris 10 x86 pass regression tests fine when I force the use of new snprintf(). The problem should be win32 - specific. I will investigate it throughly tonight. Can someone experienced in win32 what can possibly be the problem? Do we have any idea about what format string causes the regression failure? It may be that certain integer types are not promoted uniformly when pushed on the stack. Nick On Sun, 27 Feb 2005 19:07:16 +0100, Magnus Hagander [EMAIL PROTECTED] wrote: Hi! The new snpritnf code appears not to deal with 64-bit ints. I'm getting failures on win32 for int8 as well as all the time related tests (win32 uses int8 for tinmestamps). Removing the snprintf code and falling back to the OS code makes everything pass again. I would guess this affects int8 etc on other platforms as well (assuming they use our snprintf and not the libc one), but I haven't checked it. //Magnus ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] snprintf causes regression tests to fail
After some extensive debugging with Magnus's
help we finally managed to a kind of isolate the
problem. We placed snprintf.c in a separate
file, added necessary #includes and wrote
a simple main() function:
main()
{
unsigned long long ull=4567890123456789ULL;
static char buf[1024];
mysnprintf(buf,1024,%lld\n,ull);
printf(buf);
}
When compiled with -D HAVE_LONG_LONG_INT_64=1
which declares long_long and ulong_long like:
typedef long long long_long;
typedef unsigned long long ulong_long;
It compiles fine and produces desired result. If not,
it produces -869367531 as in regression tests.
Amazingly enough HAVE_LONG_LONG_INT_64 is
defined when compilation comes to src/port/snprintf.c
but the result is still wrong. I looked into configure.in
but the check for HAVE_LONG_LONG_INT_64 is too
complicated for me to understand. Bruce, could you
take a look at this? I am 90% sure it is an issue with
some configure definitions.
Best regards,
Nicolai
On Mon, 28 Feb 2005 19:58:15 +0200, Nicolai Tufar [EMAIL PROTECTED] wrote:
Regression test diff is attached.
It fails on the following tests:
int8
subselect
union
sequence
It fails to display correctly number 4567890123456789.
In output is shows -869367531. Apparent overflow or
interpreting int8 as int4.
while rewriting snprintf() I did not touch the actual functions
that convert number to ASCII digit string. In truth, if you
force PostgreSQL to use snprintf.c without my patch applied
it produces the same errors.
What can be wrong? GCC bug? The one I use is:
gcc.exe (GCC) 3.3.1 (mingw special 20030804-1)
Any thoughts?
On Mon, 28 Feb 2005 09:17:20 -0500 (EST), Bruce Momjian
pgman@candle.pha.pa.us wrote:
Nicolai Tufar wrote:
Linux and Solaris 10 x86 pass regression tests fine when I force the use
of new
snprintf(). The problem should be win32 - specific. I will
investigate it throughly
tonight. Can someone experienced in win32 what can possibly be the
problem?
Yea, I am confused too because my BSD uses the new snprintf.c code was
well. Magnus, what failures are you seeing on Win32?
--
Bruce Momjian| http://candle.pha.pa.us
pgman@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup.| Newtown Square, Pennsylvania 19073
---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faq
Re: [HACKERS] snprintf causes regression tests to fail
[EMAIL PROTECTED] writes: Do we have any idea about what format string causes the regression failure? I'll bet the problem is that configure.in is doing things in the wrong order: it computes INT64_FORMAT against the system printf before deciding we should use our own printf. regards, tom lane ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] snprintf causes regression tests to fail
Nicolai Tufar wrote:
Regression test diff is attached.
It fails on the following tests:
int8
subselect
union
sequence
It fails to display correctly number 4567890123456789.
In output is shows -869367531. Apparent overflow or
interpreting int8 as int4.
while rewriting snprintf() I did not touch the actual functions
that convert number to ASCII digit string. In truth, if you
force PostgreSQL to use snprintf.c without my patch applied
it produces the same errors.
What can be wrong? GCC bug? The one I use is:
gcc.exe (GCC) 3.3.1 (mingw special 20030804-1)
I can confirm your failure in current sources on Win32:
template1=# create table test(x int8);
CREATE TABLE
template1=# insert into test values ('4567890123456789');
INSERT 17235 1
template1=# select * from test;
x
-869367531
(1 row)
and it knows it is a large number:
template1=# select * from test where x 1000::int8;
x
-869367531
(1 row)
template1=# select * from test where x 1000::int8;
x
---
(0 rows)
I am going to add some debugs to see what is being passed to *printf().
--
Bruce Momjian| http://candle.pha.pa.us
pgman@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup.| Newtown Square, Pennsylvania 19073
---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faq
Re: [HACKERS] snprintf causes regression tests to fail
Bruce Momjian wrote:
I can confirm your failure in current sources on Win32:
template1=# create table test(x int8);
CREATE TABLE
template1=# insert into test values ('4567890123456789');
INSERT 17235 1
template1=# select * from test;
x
-869367531
(1 row)
and it knows it is a large number:
template1=# select * from test where x 1000::int8;
x
-869367531
(1 row)
template1=# select * from test where x 1000::int8;
x
---
(0 rows)
I am going to add some debugs to see what is being passed to *printf().
I have not found the solution yet but am heading to bed. My next guess
is that Win32 isn't handling va_arg(..., long long int) properly.
--
Bruce Momjian| http://candle.pha.pa.us
pgman@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup.| Newtown Square, Pennsylvania 19073
---(end of broadcast)---
TIP 8: explain analyze is your friend
