Re: [Bug-wget] Wget keeps crashing on me

[William Higgs]

Ok.  I will just use the build you sent me.  Its kind of a pet-peeve of mine
to not use the most up-to-date version of a piece of software, and I am not
so advanced in my IT skills (yet) to be able to compile my own Windows
binary of wget, and I certainly don't want to compromise my system, so I
will just make do.  Thank you gentlemen for you assistance in this matter.

-Original Message-
From: Eli Zaretskii [mailto:e...@gnu.org] 
Sent: Sunday, May 14, 2017 11:02 PM
To: William Higgs <whiggs.it...@gmail.com>
Cc: tim.rueh...@gmx.de; bug-wget@gnu.org; dar...@gmail.com
Subject: Re: [Bug-wget] Wget keeps crashing on me

> From: William Higgs <whiggs.it...@gmail.com>
> Cc: <dar...@gmail.com>,
>   'Jernej Simončič' <jernej|s-w...@eternallybored.org>
> Date: Sun, 14 May 2017 21:17:02 -0400
> 
> Hey guys.  So while I was doing some research, I found the following 
> post located at
> https://stackoverflow.com/questions/35004832/wget-exe-for-windows-10/3
> 796296
> 5#37962965
> :
> "eternallybored build will crash when you are downloading a large file.
> This can be avoided by disabling LFH (Low Fragmentation Heap) by 
> GlobalFlag registry."

Makes absolutely no sense to me.  LFH is the default heap allocation
strategy on MS-Windows since Vista; disabling it is only justified when
running a program under a debugger.  Disabling LFH globally for your entire
system means you risk running out of heap memory in some memory-intensive
applications, utterly unrelated to wget.

If that particular build of wget crashes when LFH is in use, it most
probably means a subtle memory-allocation bug, which is simply swept under
the carpet by changing the algorithm for heap allocation.  So I would
suggest to simply switch to a different build of wget, instead of
compromising your entire system.

> However, after looking into how to do this, I cannot find an 
> explanation as to how to do this.  Can someone please provide some
assistance?

 
https://support.microsoft.com/en-us/help/929136/why-the-low-fragmentation-he
ap-lfh-mechanism-may-be-disabled-on-some-computers-that-are-running-windows-
server-2003,-windows-xp,-or-windows-2000

But I'm not sure this will work on Windows 10, and I urge you not to do this
in the first place.

Re: [Bug-wget] Wget keeps crashing on me

[Eli Zaretskii]

> From: William Higgs 
> Cc: ,
>   'Jernej Simončič' 
> Date: Sun, 14 May 2017 21:17:02 -0400
> 
> Hey guys.  So while I was doing some research, I found the following post
> located at
> https://stackoverflow.com/questions/35004832/wget-exe-for-windows-10/3796296
> 5#37962965
> :
> "eternallybored build will crash when you are downloading a large file.
> This can be avoided by disabling LFH (Low Fragmentation Heap) by GlobalFlag
> registry."

Makes absolutely no sense to me.  LFH is the default heap allocation
strategy on MS-Windows since Vista; disabling it is only justified
when running a program under a debugger.  Disabling LFH globally for
your entire system means you risk running out of heap memory in some
memory-intensive applications, utterly unrelated to wget.

If that particular build of wget crashes when LFH is in use, it most
probably means a subtle memory-allocation bug, which is simply swept
under the carpet by changing the algorithm for heap allocation.  So I
would suggest to simply switch to a different build of wget, instead
of compromising your entire system.

> However, after looking into how to do this, I cannot find an explanation as
> to how to do this.  Can someone please provide some assistance?

  
https://support.microsoft.com/en-us/help/929136/why-the-low-fragmentation-heap-lfh-mechanism-may-be-disabled-on-some-computers-that-are-running-windows-server-2003,-windows-xp,-or-windows-2000

But I'm not sure this will work on Windows 10, and I urge you not to
do this in the first place.

Re: [Bug-wget] Wget keeps crashing on me

[William Higgs]

Hey guys.  So while I was doing some research, I found the following post
located at
https://stackoverflow.com/questions/35004832/wget-exe-for-windows-10/3796296
5#37962965
:
"eternallybored build will crash when you are downloading a large file.
This can be avoided by disabling LFH (Low Fragmentation Heap) by GlobalFlag
registry."

However, after looking into how to do this, I cannot find an explanation as
to how to do this.  Can someone please provide some assistance?

-Original Message-
From: Tim Rühsen [mailto:tim.rueh...@gmx.de] 
Sent: Sunday, May 14, 2017 3:38 PM
To: bug-wget@gnu.org; Eli Zaretskii <e...@gnu.org>
Cc: whiggs.it...@gmail.com; dar...@gmail.com; Jernej Simončič
<jernej|s-w...@eternallybored.org>
Subject: Re: [Bug-wget] Wget keeps crashing on me

On Sonntag, 14. Mai 2017 21:34:22 CEST Eli Zaretskii wrote:
> > From: Tim Rühsen <tim.rueh...@gmx.de>
> > Cc: William Higgs <whiggs.it...@gmail.com>, 'Darshit Shah'
> > <dar...@gmail.com>, Eli Zaretskii <e...@gnu.org> Date: Sun, 14 May 2017
> > 20:17:29 +0200
> > 
> > @Eli Mybe you could test latest master without a debugger as well !?
> 
> You mean Jernej, not Eli, right?

Oh, sorry. Yeah, I meant Jernej.

Regards, Tim

Re: [Bug-wget] Wget keeps crashing on me

[Jernej Simončič]

On Sunday, May 14, 2017, 20:19:16, Eli Zaretskii wrote:

> Is that binary also of wget 1.19.1?  If so, perhaps there's a bug in
> that particular version.

The crashes started with version 1.17 IIRC.

> Also, is your build with libssl or libgnutls?  Does it support ipv6?

I use OpenSSL, 1.19.1 is built with 1.1.0e. Here's the full --version
output:

,-
| GNU Wget 1.19.1 built on mingw32.
| 
| -cares +digest -gpgme +https +ipv6 +iri +large-file +metalink -nls
| +ntlm +opie -psl +ssl/openssl
| 
| Wgetrc:
| /win32dev/misc/wget/out64/etc/wgetrc (system)
| Compile:
| x86_64-w64-mingw32-gcc -DHAVE_CONFIG_H
| -DSYSTEM_WGETRC="/win32dev/misc/wget/out64/etc/wgetrc"
| -DLOCALEDIR="/win32dev/misc/wget/out64/share/locale" -I. -I../lib
| -I../lib -I/win32dev/misc/wget/out64/include
| -I/win32dev/misc/wget/out64/include
| -I/win32dev/misc/wget/out64/include -DHAVE_LIBSSL
| -I/win32dev/misc/wget/out2/include -DNDEBUG
| Link:
| x86_64-w64-mingw32-gcc -I/win32dev/misc/wget/out64/include
| -I/win32dev/misc/wget/out64/include -DHAVE_LIBSSL
| -I/win32dev/misc/wget/out2/include -DNDEBUG
| -L/win32dev/misc/wget/out64/lib -lidn2
| -L/win32dev/misc/wget/out64/lib -lmetalink -lunistring -liconv
| -L/win32dev/misc/wget/out64/lib -lssl -lcrypto
| -L/win32dev/misc/wget/out2/lib -lz -lws2_32 ftp-opie.o mswindows.o
| openssl.o http-ntlm.o ../lib/libgnu.a -lws2_32 -lws2_32 -lws2_32
| -lws2_32 /win32dev/misc/wget/out64/lib/libiconv.a
| /win32dev/misc/wget/out64/lib/libunistring.a
| /win32dev/misc/wget/out64/lib/libiconv.a -lws2_32
| 
| Copyright (C) 2015 Free Software Foundation, Inc.
| License GPLv3+: GNU GPL version 3 or later
| .
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
| 
| Originally written by Hrvoje Niksic .
| Please send bug reports and questions to .
`-

I build everything with static libraries, to have a wget.exe that
doesn't need additional DLLs. My binary also includes 2 patches that
aren't part of wget:
https://eternallybored.org/misc/wget/src/taskbar-progress.patch
https://eternallybored.org/misc/wget/src/openssl-windows-cert-store.patch

-- 
< Jernej Simončič ><><><><>< http://eternallybored.org/ >

Never buy a car that has a wick.
   -- Ford Pinto Rule

Re: [Bug-wget] Wget keeps crashing on me

[Jernej Simončič]

On Sunday, May 14, 2017, 19:46:45, Darshit Shah wrote:

> Then could you please provide us with a command line that sometimes
> crashes  and preferably the debug output from Wget when it crashes. Not the
> debugger, but just the debug.

I use either

  wget -t0 -c -S 

or

  wget -t0 -c -S --trust-server-names 

Seems to happen more often with larger files. I don't remember any
special output when the crash happens - just the standard progress
bar, and Windows' crash notification.

> I would like to reduce it to some kind of a minimal test case. Maybe the
> progress bar routines are crashing, or maybe it is something to do with how
> we access the filesystem, or something entirely different.

I thought about the progress bar, since it does behave a bit weird on
Windows - specifically, if the filename is scrolling, once it starts
off-screen, the last letter keeps getting repeated, while the rest of
the progress bar stops updating, however the crashes seem to happen
regardless of how the filename is displayed.

> However, since you say you can't reproduce it inside a debugger, I would
> hazard a guess that the issue lies somewhere in using uninitialized memory.
> The debugger will initialize the memory and hence prevent some class of
> bugs from occurring. An audit of the WIndows-specific code might be in
> order for this since I haven't heard of such an issue on *nix machines.

The annoying thing is that I haven't even had a crash since I set up
DrMingw as just-in-time debugger (debugger that Windows invokes only
when a crash occurs, and which otherwise isn't running). The last time
my build crashed for me was about a month ago, downloading some ISO
file at work, while I hadn't had it crash on my home computer in well
over a year (I even tried setting up a batch file that just ran wget
in a loop and leaving it in the background, but it wouldn't crash).

-- 
< Jernej Simončič ><><><><>< http://eternallybored.org/ >

The bus that left the stop just before you got there is your bus.
   -- First Law of Bus Transportation

Re: [Bug-wget] Wget keeps crashing on me

[William Higgs]

But looking at Wget's FAQ page, the source that the binary was obtained from is 
listed in the FAQ as a place to obtain a Windows binary of the wget.  So is 
that no longer applicable?  Do I need to use the 32 bit version instead?

-Original Message-
From: Tim Rühsen [mailto:tim.rueh...@gmx.de] 
Sent: Sunday, May 14, 2017 2:17 PM
To: bug-wget@gnu.org
Cc: William Higgs <whiggs.it...@gmail.com>; 'Darshit Shah' <dar...@gmail.com>; 
Eli Zaretskii <e...@gnu.org>
Subject: Re: [Bug-wget] Wget keeps crashing on me

On Sonntag, 14. Mai 2017 13:21:44 CEST William Higgs wrote:
> Attached is the output from “—version” argument.  Running debug on the 
> commands to see what the issue is now.  Will email results.

Hi William,

just an idea... recently we fixed two issues with idn_free2() that could cause 
crashes. Could you either build without IRI support and test again (does the 
issue still occur ?) or build using the latest git master branch ?

@Eli Mybe you could test latest master without a debugger as well !?

With Best Regards, tim

Re: [Bug-wget] Wget keeps crashing on me

[William Higgs]

Also found this when I scrolled down the page a bit:



 

-Original Message-
From: Tim Rühsen [mailto:tim.rueh...@gmx.de] 
Sent: Sunday, May 14, 2017 2:17 PM
To: bug-wget@gnu.org
Cc: William Higgs <whiggs.it...@gmail.com>; 'Darshit Shah' <dar...@gmail.com>; 
Eli Zaretskii <e...@gnu.org>
Subject: Re: [Bug-wget] Wget keeps crashing on me

 

On Sonntag, 14. Mai 2017 13:21:44 CEST William Higgs wrote:

> Attached is the output from “—version” argument.  Running debug on the 

> commands to see what the issue is now.  Will email results.

 

Hi William,

 

just an idea... recently we fixed two issues with idn_free2() that could cause 
crashes. Could you either build without IRI support and test again (does the 
issue still occur ?) or build using the latest git master branch ?

 

@Eli Mybe you could test latest master without a debugger as well !?

 

With Best Regards, tim

Re: [Bug-wget] Wget keeps crashing on me

[Tim Rühsen]

On Sonntag, 14. Mai 2017 21:34:22 CEST Eli Zaretskii wrote:
> > From: Tim Rühsen 
> > Cc: William Higgs , 'Darshit Shah'
> > , Eli Zaretskii  Date: Sun, 14 May 2017
> > 20:17:29 +0200
> > 
> > @Eli Mybe you could test latest master without a debugger as well !?
> 
> You mean Jernej, not Eli, right?

Oh, sorry. Yeah, I meant Jernej.

Regards, Tim

signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Wget keeps crashing on me

[Darshit Shah]

Then could you please provide us with a command line that sometimes
crashes  and preferably the debug output from Wget when it crashes. Not the
debugger, but just the debug.

I would like to reduce it to some kind of a minimal test case. Maybe the
progress bar routines are crashing, or maybe it is something to do with how
we access the filesystem, or something entirely different.
However, since you say you can't reproduce it inside a debugger, I would
hazard a guess that the issue lies somewhere in using uninitialized memory.
The debugger will initialize the memory and hence prevent some class of
bugs from occurring. An audit of the WIndows-specific code might be in
order for this since I haven't heard of such an issue on *nix machines.

On 14 May 2017 at 19:30, Jernej Simončič 
wrote:

> On Sunday, May 14, 2017, 19:07:49, Darshit Shah wrote:
>
> > I'm not sure what the exact issue here is. But it indeed seems like the
> > issue lies somewhere in how your particular binary of Wget was built. I
> > can't reproduce the issue on my Linux system, and as Eli says he can't
> > reproduce it with his binaries either. Of course, he has linked to a much
> > older version of Wget, but if I understand right, he even has newer
> builds
> > available.
>
> The wget binary I have on my site
> (https://eternallybored.org/misc/wget/) sometimes crashes when
> finishing download, but I haven't been able to reproduce the crash
> under a debugger (and I haven't had it crash since I installed DrMingw
> as JIT debugger either), so I'm not sure where the problem is.
>
> --
> < Jernej Simončič ><><><><>< http://eternallybored.org/ >
>
> It's better to be rich and healthy than poor and sick.
>-- Carson's Law
>
>
>


-- 
Thanking You,
Darshit Shah

Re: [Bug-wget] Wget keeps crashing on me

[Jernej Simončič]

On Sunday, May 14, 2017, 19:07:49, Darshit Shah wrote:

> I'm not sure what the exact issue here is. But it indeed seems like the
> issue lies somewhere in how your particular binary of Wget was built. I
> can't reproduce the issue on my Linux system, and as Eli says he can't
> reproduce it with his binaries either. Of course, he has linked to a much
> older version of Wget, but if I understand right, he even has newer builds
> available.

The wget binary I have on my site
(https://eternallybored.org/misc/wget/) sometimes crashes when
finishing download, but I haven't been able to reproduce the crash
under a debugger (and I haven't had it crash since I installed DrMingw
as JIT debugger either), so I'm not sure where the problem is.

-- 
< Jernej Simončič ><><><><>< http://eternallybored.org/ >

It's better to be rich and healthy than poor and sick.
   -- Carson's Law

Re: [Bug-wget] Wget keeps crashing on me

[Eli Zaretskii]

> From: William Higgs 
> Date: Sun, 14 May 2017 12:13:58 -0400
> 
> So just to be clear, you want me to use an older release of wget?

No, I'm just saying that the version I built worked without crashing.
You may wish to try it; if it works on your system, it might mean the
problem is not with the OS, but with the wget build you have.

Re: [Bug-wget] Wget keeps crashing on me

[Eli Zaretskii]

> From: William Higgs 
> Date: Sun, 14 May 2017 10:27:12 -0400
> 
> And I saw that you had stated that it was working on Windows 7, which
> further convinces me that it is probably a windows 10 thing.  I originally
> thought this was the case because, while the faulting application is wget,
> the faulting module (module I assume to mean what actually caused the crash
> in the application), is ntdll.dll, which is a core system dll.  But sfc
> scans return no issues..

I'm not sure this is a Windows problem.  Crashes inside system DLLs
more often than not are caused by bugs in the applications.

Re: [Bug-wget] Wget keeps crashing on me

[Eli Zaretskii]

> From: William Higgs 
> Date: Sun, 14 May 2017 10:23:35 -0400
> 
> The txt file contains the output from the command "wget --version".

Maybe I'm missing something, but I don't see that.  All I see is this:

  Description   : Faulting application name: wget.exe, version: 0.0.0.0, 
time stamp: 0x003cc610

which doesn't show the wget version.

I tried with wget 1.16.1, which I built myself.  You can find its
binaries here:

  
https://sourceforge.net/projects/ezwinports/files/wget-1.16.1-w32-bin.zip/download

> As for the second question, I obtained the binaries by utilizing
> chocolatey's package management system.  If you are more familiar
> with Linux, you can think of chocolatey as the official unofficial
> (while not directly supported by Microsoft, Microsoft has
> incorporated its use into Powershell 5's package management cmdlets)
> "apt-get" for Windows (https://chocolatey.org/).

Thanks for the info.

P.S. Please keep the list address on the CC.

Re: [Bug-wget] Wget keeps crashing on me

[Eli Zaretskii]

> From: William Higgs 
> Date: Sat, 13 May 2017 17:37:13 -0400
> 
> So this may have nothing to do with wget (probably very likely, as Windows
> 10 creators update continues to be a very large thorn in my side), but wget
> keeps crashing when I run the attached bat file (converted to txt).  I saved
> the event logs associated with the crash, but again, it looks more like an
> os issue than wget.  Still, wanted to get your opinion on the matter.  Also,
> thanks for the awesome, quality, free software.

FWIW, this works for me, on Windows 7.

What version of wget did you use, and where did you get the binaries?