Re: pg_receivewal.exe unhandled exception in zlib1.dll
Please excuse my late reply. On Wed, Feb 16, 2022 at 12:20 PM Juan José Santamaría Flecha < juanjo.santama...@gmail.com> wrote: > On Tue, Feb 15, 2022 at 5:25 PM Andres Freund wrote: > >> >> FWIW, I've looked at using either vcpkg or conan to more easily >> install / >> > build dependencies on windows, in the right debug / release mode. >> >> The former was easier, but unfortunately the zlib, lz4, ... packages don't >> include the gzip, lz4 etc binaries right now. That might be easy enough >> to fix >> with an option. It does require building the dependencies locally. >> >> Conan seemed to be a nicer architecture, but there's no builtin way to >> update >> to newer dependency versions, and non-versioned dependencies don't >> actually >> work. It has pre-built dependencies for the common cases. >> >> I will have to do some testing with Conan and see how both compare. > I have found a couple of additional issues when trying to build using Conan, all libraries are static and there aren't any prebuilt packages for MSVC 2022 yet. Also for some specific packages: - zstd doesn't include the binaries either. - gettext packages only provides binaries, no libraries, nor headers. - openssl builds the binaries, but is missing some objects in the libraries (error LNK2019: unresolved external symbol). - libxml builds the binaries, but is missing some objects in the libraries (error LNK2019: unresolved external symbol). ,Regards, Juan José Santamaría Flecha
Re: pg_receivewal.exe unhandled exception in zlib1.dll
On Tue, Feb 15, 2022 at 5:25 PM Andres Freund wrote: > > > If this is something we want to discard altogether, we can check at build > > time with 'dumpbin' to ensure that all dlls share the same vcruntime.dll. > > I think these days it's mostly a question of ucrtbase.dll vs ucrtbased.dll, > right? > > Either one would do the trick, just to tell apart 'debug' from 'release'. > FWIW, I've looked at using either vcpkg or conan to more easily install > / > build dependencies on windows, in the right debug / release mode. > > The former was easier, but unfortunately the zlib, lz4, ... packages don't > include the gzip, lz4 etc binaries right now. That might be easy enough to > fix > with an option. It does require building the dependencies locally. > > Conan seemed to be a nicer architecture, but there's no builtin way to > update > to newer dependency versions, and non-versioned dependencies don't actually > work. It has pre-built dependencies for the common cases. > > I find that the repositories that are being currently proposed as a source for the Windows depencies [1], will lead to outdated or stale packages in many cases. I have been testing vcpkg, and my only grievances have been that some packages were terribly slow to install and a couple were missing: Kerberos (but its own project seems to be active [2]) and ossp-uuid (which is to be expected). I will have to do some testing with Conan and see how both compare. [1] https://www.postgresql.org/docs/devel/install-windows-full.html [2] https://web.mit.edu/Kerberos/ Regards, Juan José Santamaría Flecha
Re: pg_receivewal.exe unhandled exception in zlib1.dll
Hi, On 2022-02-15 11:21:42 +0100, Juan José Santamaría Flecha wrote: > Building with a mix of debug and release components is not a good practice > for issues like this, but is not something that MSVC forbids. It's not a problem if you never need share memory, file descriptors, locales, ... state across DLL boundaries... > If this is something we want to discard altogether, we can check at build > time with 'dumpbin' to ensure that all dlls share the same vcruntime.dll. I think these days it's mostly a question of ucrtbase.dll vs ucrtbased.dll, right? FWIW, I've looked at using either vcpkg or conan to more easily install / build dependencies on windows, in the right debug / release mode. The former was easier, but unfortunately the zlib, lz4, ... packages don't include the gzip, lz4 etc binaries right now. That might be easy enough to fix with an option. It does require building the dependencies locally. Conan seemed to be a nicer architecture, but there's no builtin way to update to newer dependency versions, and non-versioned dependencies don't actually work. It has pre-built dependencies for the common cases. Greetings, Andres Freund
Re: pg_receivewal.exe unhandled exception in zlib1.dll
On Mon, Feb 14, 2022 at 7:31 AM Michael Paquier wrote: > On Fri, Feb 11, 2022 at 07:50:44AM -0800, Andres Freund wrote: > > > I hit this as well. The problem is really caused by using a debug build > of > > postgres vs a production build of zlib. The use different C runtimes, > with > > different file descriptors. A lot of resources in the windows world are > > unfortunately tied to the C runtime and that there can multiple C > runtimes in > > a single process. > > It may be worth warning about that at build time, or just document the > matter perhaps? I don't recall seeing any MSIs related to zlib that > have compile with the debug options. > Building with a mix of debug and release components is not a good practice for issues like this, but is not something that MSVC forbids. If this is something we want to discard altogether, we can check at build time with 'dumpbin' to ensure that all dlls share the same vcruntime.dll. Regards, Juan José Santamaría Flecha
Re: pg_receivewal.exe unhandled exception in zlib1.dll
On Fri, Feb 11, 2022 at 07:50:44AM -0800, Andres Freund wrote: > On 2022-02-11 16:33:11 +0100, Juan José Santamaría Flecha wrote: >> The problem comes from the file descriptor passed to gzdopen() in >> 'src/bin/pg_basebackup/walmethods.c'. Using gzopen() instead, solves the >> issue without ifdefing for WIN32. Please find attached a patch for so. This looks wrong to me as gzclose() would close() the file descriptor we pass in via gzdopen(), and some code paths of walmethods.c rely on this assumption so this would leak fds on repeated errors. > I hit this as well. The problem is really caused by using a debug build of > postgres vs a production build of zlib. The use different C runtimes, with > different file descriptors. A lot of resources in the windows world are > unfortunately tied to the C runtime and that there can multiple C runtimes in > a single process. It may be worth warning about that at build time, or just document the matter perhaps? I don't recall seeing any MSIs related to zlib that have compile with the debug options. -- Michael signature.asc Description: PGP signature
Re: pg_receivewal.exe unhandled exception in zlib1.dll
Hi, On 2022-02-11 16:33:11 +0100, Juan José Santamaría Flecha wrote: > If you execute pg_receivewal.exe with the option "--compression-method > gzip" it will fail with no error. You can see an error in the db log: > > 2022-02-10 11:46:32.725 CET [11664][walsender] [pg_receivewal][3/0:0] LOG: > could not receive data from client: An existing connection was forcibly > closed by the remote host. > > Tracing the execution you can see: > > Unhandled exception at 0x7FFEA78C1208 (ucrtbase.dll) in An invalid > parameter was passed to a function that considers invalid parameters fatal. > ucrtbase.dll!7ff8e8ae36a2() Unknown > > zlib1.dll!gz_comp(gz_state * state, int flush) Line 111 C > zlib1.dll!gz_write(gz_state * state, const void * buf, unsigned __int64 > len) Line 235 C > pg_receivewal.exe!dir_write(void * f, const void * buf, unsigned __int64 > count) Line 300 C > pg_receivewal.exe!ProcessXLogDataMsg(pg_conn * conn, StreamCtl * stream, > char * copybuf, int len, unsigned __int64 * blockpos) Line 1150 C > pg_receivewal.exe!HandleCopyStream(pg_conn * conn, StreamCtl * stream, > unsigned __int64 * stoppos) Line 850 C > pg_receivewal.exe!ReceiveXlogStream(pg_conn * conn, StreamCtl * stream) > Line 605 C > pg_receivewal.exe!StreamLog() Line 636 C > pg_receivewal.exe!main(int argc, char * * argv) Line 1005 C > > The problem comes from the file descriptor passed to gzdopen() in > 'src/bin/pg_basebackup/walmethods.c'. Using gzopen() instead, solves the > issue without ifdefing for WIN32. Please find attached a patch for so. I hit this as well. The problem is really caused by using a debug build of postgres vs a production build of zlib. The use different C runtimes, with different file descriptors. A lot of resources in the windows world are unfortunately tied to the C runtime and that there can multiple C runtimes in a single process. Greetings, Andres Freund
pg_receivewal.exe unhandled exception in zlib1.dll
Hello,
If you execute pg_receivewal.exe with the option "--compression-method
gzip" it will fail with no error. You can see an error in the db log:
2022-02-10 11:46:32.725 CET [11664][walsender] [pg_receivewal][3/0:0] LOG:
could not receive data from client: An existing connection was forcibly
closed by the remote host.
Tracing the execution you can see:
Unhandled exception at 0x7FFEA78C1208 (ucrtbase.dll) in An invalid
parameter was passed to a function that considers invalid parameters fatal.
ucrtbase.dll!7ff8e8ae36a2() Unknown
> zlib1.dll!gz_comp(gz_state * state, int flush) Line 111 C
zlib1.dll!gz_write(gz_state * state, const void * buf, unsigned __int64
len) Line 235 C
pg_receivewal.exe!dir_write(void * f, const void * buf, unsigned __int64
count) Line 300 C
pg_receivewal.exe!ProcessXLogDataMsg(pg_conn * conn, StreamCtl * stream,
char * copybuf, int len, unsigned __int64 * blockpos) Line 1150 C
pg_receivewal.exe!HandleCopyStream(pg_conn * conn, StreamCtl * stream,
unsigned __int64 * stoppos) Line 850 C
pg_receivewal.exe!ReceiveXlogStream(pg_conn * conn, StreamCtl * stream)
Line 605 C
pg_receivewal.exe!StreamLog() Line 636 C
pg_receivewal.exe!main(int argc, char * * argv) Line 1005 C
The problem comes from the file descriptor passed to gzdopen() in
'src/bin/pg_basebackup/walmethods.c'. Using gzopen() instead, solves the
issue without ifdefing for WIN32. Please find attached a patch for so.
Regards,
Juan José Santamaría Flecha
diff --git a/src/bin/pg_basebackup/walmethods.c b/src/bin/pg_basebackup/walmethods.c
index a6d08c1..bc0790a 100644
--- a/src/bin/pg_basebackup/walmethods.c
+++ b/src/bin/pg_basebackup/walmethods.c
@@ -143,7 +143,7 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
#ifdef HAVE_LIBZ
if (dir_data->compression_method == COMPRESSION_GZIP)
{
- gzfp = gzdopen(fd, "wb");
+ gzfp = gzopen(tmppath, "wb");
if (gzfp == NULL)
{
dir_data->lasterrno = errno;
