Re: wget crash

[Steven Enderle]

Great!

Thanks a lot for your help

Regards

Steven Enderle

Hrvoje Niksic wrote:
> Hack Kampbjørn <[EMAIL PROTECTED]> writes:
> 
> 
>>>assertion "percentage <= 100" failed: file "progress.c", line 552
>>>zsh: abort (core dumped)  wget -m -c --tries=0
>>>ftp://ftp.scene.org/pub/music/artists/nutcase/mp3/timeofourlives.mp3
>>
>>progress.c
>>  int percentage = (int)(100.0 * size / bp->total_length);
>>
>>  assert (percentage <= 100);
>>Of course the assert will fail, size is bigger than total_length !
> 
> [...]
> 
>>To reproduce with wget-1.8.1
>>$ wget ftp://sunsite.dk/disk1/gnu/wget/wget-1.8{,.1}.tar.gz
>>$ cat wget-1.8.tar.gz >> wget-1.8.1.tar.gz
>>$ wget -d -c ftp://sunsite.dk/disk1/gnu/wget/wget-1.8.1.tar.gz
> 
> 
> Thanks for looking into this.  There are two problems here, and most
> likely two separate bugs.
> 
> First, I cannot repeat your test case.  Maybe sunsite.dk changed their
> FTP server since Feb 15; anyway, what I get is:
> 
> --> REST 2185627
> 
> 350 Restarting at 2185627
> --> RETR wget-1.8.1.tar.gz
> 
> 451-Restart offset 2185627 is too large for file size 1097780.
> 451 Restart offset reset to 0
> 
> Wget (bogusly) considers the 451 response to be "error in server
> response" and retries.  That's bug number one, but it also means that
> I cannot repeat your test case.
> 
> 
> Bug number two is the one the reporter saw.  At first I didn't quite
> understand how it can happen, since bar_update() explicitly guards
> against such a condition:
> 
>   if (bp->total_length > 0
>   && bp->count + bp->initial_length > bp->total_length)
> /* We could be downloading more than total_length, e.g. when the
>server sends an incorrect Content-Length header.  In that case,
>adjust bp->total_length to the new reality, so that the code in
>create_image() that depends on total size being smaller or
>equal to the expected size doesn't abort.  */
> bp->total_length = bp->count + bp->initial_length;
> 
> The problem is that the same guard is not implemented in bar_create()
> and bar_finish(), which also call create_image().  In the FTP case,
> the crash comes from bar_create.  This patch should fix it.
> 
> 2002-04-11  Hrvoje Niksic  <[EMAIL PROTECTED]>
> 
>   * progress.c (bar_create): If INITIAL is larger than TOTAL, fix
>   TOTAL.
>   (bar_finish): Likewise.
> 
> Index: src/progress.c
> ===
> RCS file: /pack/anoncvs/wget/src/progress.c,v
> retrieving revision 1.27
> diff -u -r1.27 progress.c
> --- src/progress.c2002/04/11 17:49:32 1.27
> +++ src/progress.c2002/04/11 18:49:08
> @@ -461,6 +461,11 @@
>  
>memset (bp, 0, sizeof (*bp));
>  
> +  /* In theory, our callers should take care of this pathological
> + case, but it can sometimes happen. */
> +  if (initial > total)
> +total = initial;
> +
>bp->initial_length = initial;
>bp->total_length   = total;
>  
> @@ -493,7 +498,7 @@
> adjust bp->total_length to the new reality, so that the code in
> create_image() that depends on total size being smaller or
> equal to the expected size doesn't abort.  */
> -bp->total_length = bp->count + bp->initial_length;
> +bp->total_length = bp->initial_length + bp->count;
>  
>/* This code attempts to determine the current download speed.  We
>   measure the speed over the interval of approximately three
> @@ -564,6 +569,11 @@
>  bar_finish (void *progress, long dltime)
>  {
>struct bar_progress *bp = progress;
> +
> +  if (bp->total_length > 0
> +  && bp->count + bp->initial_length > bp->total_length)
> +/* See bar_update() for explanation. */
> +bp->total_length = bp->initial_length + bp->count;
>  
>create_image (bp, dltime);
>display_image (bp->buffer);
> 
> 
> 

Re: maybe code from pavuk would help

[Noel Koethe]

On Fre, 12 Apr 2002, Hrvoje Niksic wrote:

> > I tested pavuk (http://www.pavuk.org/, GPL)
> [...]
> > Maybe there is some code which could be used in wget.:)
> > So the wheel wouldn't invented twice.
> 
> Is Pavuk's code assigned to the FSF?

afaik no.
Author: Ondrejicka Stefan <[EMAIL PROTECTED]>

There is a wget-pavuk.HOWTO in the source where the differnces
are listed.

-- 
Noèl Köthe

Re: HTTP 1.1

[csaba . raduly]

On 11/04/2002 18:26:15 hniksic wrote:

>"Boaz Yahav" <[EMAIL PROTECTED]> writes:
>
>> Is there any way to make Wget use HTTP/1.1 ?
>
>Unfortunately, no.

Sure it can be made to use HTTP 1.1

--- http.c.orig   Wed Jan 30 14:10:42 2002
+++ http.c  Fri Apr 12 11:56:22 2002
@@ -838,7 +838,7 @@
  + 64);
   /* Construct the request.  */
   sprintf (request, "\
-%s %s HTTP/1.0\r\n\
+%s %s HTTP/1.1\r\n\
 User-Agent: %s\r\n\
 Host: %s%s%s%s\r\n\
 Accept: %s\r\n\




:-)

--
Csaba Ráduly, Software Engineer   Sophos Anti-Virus
email: [EMAIL PROTECTED]http://www.sophos.com
US Support: +1 888 SOPHOS 9 UK Support: +44 1235 559933

Re: wget-1.8.1: build failure on SGI IRIX 6.5 with c89

[Ian Abbott]

On 11 Apr 2002 at 18:55, Nelson H. F. Beebe wrote:

> >> what happens if you configure it with the option
> >> --x-includes=/usr/local/include ?
> 
> On SGI IRIX 6.5, in a clean directory, I unbundled wget-1.8.1.tar.gz,
> and did this:
> 
>   % env CC=c89 ./configure --x-includes=/usr/local/include
> 
>   % grep HAVE_NLS src/config.h
>   #define HAVE_NLS 1
> 
>   % grep HAVE_LIBINTL_H src/config.h
>   /* #undef HAVE_LIBINTL_H */

Okay so --x-includes didn't achieve much. I thought the x might
stand for 'extra', but I guess it must be for the X Window System,
and therefore irrelevant to Wget.

How about:
% env CC=c89 CPPFLAGS='-I/usr/local/include' ./configure

There's got to be some way to get this thing to build!

I just tried moving libintl.h into /usr/local/include on my machine
and doing something similar:

bash$ CC=cc CPPFLAGS='-I/usr/local/include' ./configure

and it managed to set both HAVE_NLS and HAVE_LIBINTL_H in the
resulting src/config.h and it managed to build okay.

Re: wget-1.8.1: build failure on SGI IRIX 6.5 with c89

[Ian Abbott]

On 12 Apr 2002 at 12:18, Ian Abbott wrote:

> How about:
> % env CC=c89 CPPFLAGS='-I/usr/local/include' ./configure

To sum up the email correspondence between Nelson and myself since
this message, the above configure command resulted in both HAVE_NLS
and HAVE_LIBINTL_H to be defined in src/config.h and the program
built fine up until linking the program, when libtool failed to
recognize that it was given a link command due to compiler being
called 'c89'. A small patch to src/Makefile.in put that right (see
below).

There is still the question of whether the configure script and/or
src/config.h should do something about HAVE_NLS being defined, but
not HAVE_LIBINTL_H.

--- src/Makefile.in.origFri Apr 12 13:51:37 2002
+++ src/Makefile.in Fri Apr 12 13:52:09 2002
@@ -49,7 +49,7 @@
 INCLUDES = -I. -I$(srcdir) @SSL_INCLUDES@
 
 COMPILE = $(CC) $(INCLUDES) $(CPPFLAGS)  $(DEFS) $(CFLAGS)
-LINK= @LIBTOOL@ $(CC) $(CFLAGS) $(LDFLAGS) -o $@
+LINK= @LIBTOOL@ --mode=link $(CC) $(CFLAGS) $(LDFLAGS) -o $@
 INSTALL = @INSTALL@
 INSTALL_PROGRAM = @INSTALL_PROGRAM@
 RM = rm -f

Re: Referrer Faking and other nifty features

[Hrvoje Niksic]

"Dan Mahoney, System Admin" <[EMAIL PROTECTED]> writes:

>> It is the --referer option, see (wget)HTTP Options, from the Info
>> documentation.
>
> Yes, that allows me to specify _A_ referrer, like www.aol.com.  When
> I'm trying to help my users mirror their old angelfire pages or
> something like that, very often the link has to come from the same
> directory.  I'd like to see something where when wget follows a link
> to another page, or another image, it automatically supplies the URL
> of the page it followed to get there.  Is there a way to do this?

Doesn't Wget do so by default?

>> > 3) Multi-threading.
>>
>> I suppose you mean downloading several URIs in parallel.  No, wget
>> doesn't support that.  Sometimes, however, one may start several wget
>> in parallel, thanks to the shell (the & operator on Bourne shells).
>
> No, I mean downloading multiple files from the SAME uri in parallel,
> instead of downloading files one-by-one-by-one (thus saving time on
> a fast pipe).

Wget will almost certainly never be multithreaded, but I might
introduce options to make this kind of thing easier by using multiple
processes.

>> > Also, I have in the past encountered a difficulty with the ~
>> > being escaped the wrong way, has this been fixed?  I know at one
>> > point one site suggested you modify url.c to "fix" this.
>>
>> AFAIK, I have never had that problem; maybe it has been fixed.
>
> I remember the problem now.  I was trying to mirror
> homepages.go.com/~something and for whatever reason, wget would
> follow a link to homepages.go.com/~somethingelse and parse it out to
> homepages.go.com/%7esomethingelse, which for some reason the
> webserver DIDN'T like

That sounds like an extremely broken web server.  %xx has always been
a valid URL encoding.  For example, the only way to request a file
with spaces in file name is to encode spaces as %20.

Re: option -nh (was: Re: option changed: -nh -> -nH)

[Hrvoje Niksic]

Noel Koethe <[EMAIL PROTECTED]> writes:

> On Mit, 03 Apr 2002, Jens Rösner wrote:
>
>> I already complained that many old scripts now break and suggested 
>> that entering -nh at the command line would 
>> either be completely ignored or the user would be 
>> informed and wget executed nevertheless.
>> Apparently this was not regarded as useful.
>
> :( OK. So this bug wont be fixed.:(

It won't because it's not a bug.

I try not to change Wget's options gratuitously, but some amount of
change must be expected.  In this case, it really is trivial to remove
the offending `-nh'.

Re: suspected bug in WGET 1.8.1

[Hrvoje Niksic]

"Matt Jackson" <[EMAIL PROTECTED]> writes:

> I'm using the NT port of WGET 1.8.1.
>
> FTP retrieval of files works fine, retrieval of directory listings fails.
> The problem happens under certain conditions when connecting to OS2 FTP
> servers.
>
> For example, if the "current directory" on the FTP server at login time is
> "e:/abc", the command "wget ftp://userid:password@ipaddr/g:\def\test.doc";
> works fine to retrieve the file, but the command "wget
> ftp://userid:password@ipaddr/g:\def\"; fails to retrieve the directory
> listing.

Can you provide the `-d' output?  It will show the entire conversation
between Wget and the server, and thus help clarify what's going on.
Don't forget to remove the private password if it comes up in the
debug log.

Re: Referrer Faking and other nifty features

[Thomas Lussnig]

>
>
3) Multi-threading.

>>>I suppose you mean downloading several URIs in parallel.  No, wget
>>>doesn't support that.  Sometimes, however, one may start several wget
>>>in parallel, thanks to the shell (the & operator on Bourne shells).
>>>
>>No, I mean downloading multiple files from the SAME uri in parallel,
>>instead of downloading files one-by-one-by-one (thus saving time on
>>a fast pipe).
>>
>
>Wget will almost certainly never be multithreaded, but I might
>introduce options to make this kind of thing easier by using multiple
>processes.
>
Hi,
i think for this feature there is no need for Multithreading. If the 
Procedures do not use Struct.
It should be posible to handle that with select. That would mean an FIFO 
with the URL's to fetch
(think this is already there) and an list of 1-16 Handles for conections 
witch where assinged to an
function that handle them

struct {
int fd;
int wait_read;
int wait_write;
int do_Write(int fd);
int do_Read(int fd);
}

So that if one fd become -1 the "loader" take an new url and initate the 
download.

And than shedulingwould work with the select(int,) what about this 
idee ?

Cu Thomas Lußnig

newbie question

[dbotham]

Just when I thought it was safe to start downloading files, I get this:

wget --mirror -v -I/ -X/report,/Software -w1 -gon
ftp://x:[EMAIL PROTECTED]
--11:27:41--  ftp://x:[EMAIL PROTECTED]:21/
   => `64.226.243.208/.listing'
Connecting to 64.226.243.208:21... connected!
Logging in as xx... Logged in!
==> TYPE I ... done.  ==> CWD not needed.
==> PORT ... done.==> LIST ... done.

0K -> .. ..

11:27:42 (713.53 KB/s) - `64.226.243.208/.listing' saved [16805]

--11:27:43--  ftp://x:[EMAIL PROTECTED]:21/
   => `64.226.243.208/index.html'
==> CWD not required.
==> PORT ... done.==> RETR  ...
No such file `'.


FINISHED --11:27:43--
Downloaded: 0 bytes in 0 files


When I look in my local directory the olny thing I see is the '.listing'
file.  Any suggestions?

Thanks,

Dave...

Re: URI-parsing bug

[Hrvoje Niksic]

Tristan Horn <[EMAIL PROTECTED]> writes:

> tris.net/index.html: merge("http://tris.net/";, "//www.arrl.org/") ->
> http://tris.net//www.arrl.org/
> (it should return http://www.arrl.org/)
>
> See page 11 of rfc1630 and page 11 of rfc2396 for more details.  I
> may well be the only person using 'em, though... :)

Thanks for the report.  And, as it happens, you're not the only person
using them.  Here is the fix that has been in CVS for some time now:

Index: src/url.c
===
RCS file: /pack/anoncvs/wget/src/url.c,v
retrieving revision 1.67
retrieving revision 1.68
diff -u -r1.67 -r1.68
--- src/url.c   2001/12/14 15:45:59 1.67
+++ src/url.c   2002/01/14 01:56:40 1.68
@@ -1575,6 +1575,37 @@
  memcpy (constr + baselength, link, linklength);
  constr[baselength + linklength] = '\0';
}
+  else if (linklength > 1 && *link == '/' && *(link + 1) == '/')
+   {
+ /* LINK begins with "//" and so is a net path: we need to
+replace everything after (and including) the double slash
+with LINK. */
+
+ /* uri_merge("foo", "//new/bar")-> "//new/bar"  */
+ /* uri_merge("//old/foo", "//new/bar")  -> "//new/bar"  */
+ /* uri_merge("http://old/foo";, "//new/bar") -> "http://new/bar"; */
+
+ int span;
+ const char *slash;
+ const char *start_insert;
+
+ /* Look for first slash. */
+ slash = memchr (base, '/', end - base);
+ /* If found slash and it is a double slash, then replace
+from this point, else default to replacing from the
+beginning.  */
+ if (slash && *(slash + 1) == '/')
+   start_insert = slash;
+ else
+   start_insert = base;
+
+ span = start_insert - base;
+ constr = (char *)xmalloc (span + linklength + 1);
+ if (span)
+   memcpy (constr, base, span);
+ memcpy (constr + span, link, linklength);
+ constr[span + linklength] = '\0';
+   }
   else if (*link == '/')
{
  /* LINK is an absolute path: we need to replace everything

Re: Suggestions

[Hrvoje Niksic]

Fernando Cassia <[EMAIL PROTECTED]> writes:

>>   You could add function that could receive range of bytes. Ex. wget
>>   -begin 2300 -end 40 it would download file from 2300 byte to
>>   4 byte.

A patch implementing a `--range' option is pending and will be part of
Wget 1.9.

Re: Referrer Faking and other nifty features

[Ian Abbott]

On 12 Apr 2002 at 17:21, Thomas Lussnig wrote:

> So that if one fd become -1 the "loader" take an new url and initate the 
> download.
> 
> And than shedulingwould work with the select(int,) what about this 
> idee ?

It would certainly make handling the logging output a bit of a
challenge, especially the progress indication.

Re: No clobber and .shtml files

[Ian Abbott]

On 11 Apr 2002 at 21:00, Hrvoje Niksic wrote:

> This change is fine with me.  I vaguely remember that this test is
> performed in two places; you might want to create a function.

Certainly. Where's the best place for it? utils.c?

Re: No clobber and .shtml files

[Hrvoje Niksic]

"Ian Abbott" <[EMAIL PROTECTED]> writes:

> On 11 Apr 2002 at 21:00, Hrvoje Niksic wrote:
>
>> This change is fine with me.  I vaguely remember that this test is
>> performed in two places; you might want to create a function.
>
> Certainly. Where's the best place for it? utils.c?

As good a place as any.

Re: /usr/include/stdio.h:120: previous declaration of `va_list'

[Hrvoje Niksic]

Kevin Rodgers <[EMAIL PROTECTED]> writes:

> 1. Don't #define _XOPEN_SOURCE 500 (by commenting it out).
>
> 2. Do #define _VA_ALIST.
>
> I can confirm that (1) works.  I didn't try (2).

Could you please try (2) and see if it works out?

I'm reluctant to withdraw the _XOPEN_SOURCE definition because it's
supposed to create the kind of "environment" that we want --
standards-compliant with useful extensions.  Without it, some
functions we use just don't get declared.  (I think strptime is one of
them, but there are probably more.)  I'm keeping that option as a last
resort.

Thanks for the report and the analysis.

Re: newbie question

[dbotham]

All,

Found the problem.  The remote ftp server is NT and was set to use MSDOS
listing.   A!  All together
now.

Thanks,

Dave...


|-+>
| |   dbotham@edeltacom|
| |   .com |
| ||
| |   04/12/2002 11:44 |
| |   AM   |
| ||
|-+>
  
>--|
  |
  |
  |   To:   [EMAIL PROTECTED]
  |
  |   cc:  
  |
  |   Subject:  newbie question
  |
  
>--|




I am not sure if this got though the first time, so I am sending it
again... Sorry for the clutter...

Also, what I am trying to do is ftp all the files off the target server and
save them to the my local system.
Why is wget looking for 'index.html'?  I am not using http://, rather
ftp://.

Any help would be really great.

Just when I thought it was safe to start downloading files, I get this:

wget --mirror -v -I/ -X/report,/Software -w1 -gon
ftp://x:[EMAIL PROTECTED]
--11:27:41--  ftp://x:[EMAIL PROTECTED]:21/
   => `64.226.243.208/.listing'
Connecting to 64.226.243.208:21... connected!
Logging in as xx... Logged in!
==> TYPE I ... done.  ==> CWD not needed.
==> PORT ... done.==> LIST ... done.

0K -> .. ..

11:27:42 (713.53 KB/s) - `64.226.243.208/.listing' saved [16805]

--11:27:43--  ftp://x:[EMAIL PROTECTED]:21/
   => `64.226.243.208/index.html'
==> CWD not required.
==> PORT ... done.==> RETR  ...
No such file `'.


FINISHED --11:27:43--
Downloaded: 0 bytes in 0 files


When I look in my local directory the olny thing I see is the '.listing'
file.  Any suggestions?

Thanks,

Dave...

Re: No clobber and .shtml files

[Ian Abbott]

On 11 Apr 2002 at 21:00, Hrvoje Niksic wrote:

> This change is fine with me.  I vaguely remember that this test is
> performed in two places; you might want to create a function.

I've found three places where it checks the suffix, so I called a
new function in all three places for consistency. One of those
places performed a case-insensitive comparison so I made my
function do that too.

Hrvoje, you may wish to review whether checking the new extensions
in all three places (but particularly recur.c) is a good idea or
not before I commit the patch.

src/ChangeLog entry:

2002-04-12  Ian Abbott  <[EMAIL PROTECTED]>

* utils.c (has_html_suffix_p): New function to text filename for common
html extensions.

* utils.h: Declare it.

* http.c (http_loop): Use it instead of previous test.

* retr.c (retrieve_url): Ditto.

* recur.c (download_child_p): Ditto.

Index: src/http.c
===
RCS file: /pack/anoncvs/wget/src/http.c,v
retrieving revision 1.86
diff -u -r1.86 http.c
--- src/http.c  2002/04/11 17:49:32 1.86
+++ src/http.c  2002/04/12 17:35:02
@@ -1405,7 +1405,7 @@
   int use_ts, got_head = 0;/* time-stamping info */
   char *filename_plus_orig_suffix;
   char *local_filename = NULL;
-  char *tms, *suf, *locf, *tmrate;
+  char *tms, *locf, *tmrate;
   uerr_t err;
   time_t tml = -1, tmr = -1;   /* local and remote time-stamps */
   long local_size = 0; /* the size of the local file */
@@ -1465,9 +1465,8 @@
   *dt |= RETROKF;
 
   /*  Bogusness alert.  */
-  /* If its suffix is "html" or "htm", assume text/html.  */
-  if (((suf = suffix (*hstat.local_file)) != NULL)
- && (!strcmp (suf, "html") || !strcmp (suf, "htm")))
+  /* If its suffix is "html" or "htm" or similar, assume text/html.  */
+  if (has_html_suffix_p (*hstat.local_file))
*dt |= TEXTHTML;
 
   FREE_MAYBE (dummy);
Index: src/recur.c
===
RCS file: /pack/anoncvs/wget/src/recur.c,v
retrieving revision 1.43
diff -u -r1.43 recur.c
--- src/recur.c 2002/02/19 06:09:57 1.43
+++ src/recur.c 2002/04/12 17:35:02
@@ -510,7 +510,6 @@
 
   /* 6. */
   {
-char *suf;
 /* Check for acceptance/rejection rules.  We ignore these rules
for HTML documents because they might lead to other files which
need to be downloaded.  Of course, we don't know which
@@ -521,14 +520,13 @@
* u->file is not "" (i.e. it is not a directory)
and either:
  + there is no file suffix,
-+ or there is a suffix, but is not "html" or "htm",
++ or there is a suffix, but is not "html" or "htm" or similar,
 + both:
   - recursion is not infinite,
   - and we are at its very end. */
 
 if (u->file[0] != '\0'
-   && ((suf = suffix (url)) == NULL
-   || (0 != strcmp (suf, "html") && 0 != strcmp (suf, "htm"))
+   && (!has_html_suffix_p (url)
|| (opt.reclevel != INFINITE_RECURSION && depth >= opt.reclevel)))
   {
if (!acceptable (u->file))
Index: src/retr.c
===
RCS file: /pack/anoncvs/wget/src/retr.c,v
retrieving revision 1.50
diff -u -r1.50 retr.c
--- src/retr.c  2002/01/30 19:12:20 1.50
+++ src/retr.c  2002/04/12 17:35:03
@@ -384,12 +384,11 @@
 
   /* There is a possibility of having HTTP being redirected to
 FTP.  In these cases we must decide whether the text is HTML
-according to the suffix.  The HTML suffixes are `.html' and
-`.htm', case-insensitive.  */
+according to the suffix.  The HTML suffixes are `.html',
+`.htm' and a few others, case-insensitive.  */
   if (redirection_count && local_file && u->scheme == SCHEME_FTP)
{
- char *suf = suffix (local_file);
- if (suf && (!strcasecmp (suf, "html") || !strcasecmp (suf, "htm")))
+ if (has_html_suffix_p (local_file))
*dt |= TEXTHTML;
}
 }
Index: src/utils.c
===
RCS file: /pack/anoncvs/wget/src/utils.c,v
retrieving revision 1.44
diff -u -r1.44 utils.c
--- src/utils.c 2002/01/17 01:03:33 1.44
+++ src/utils.c 2002/04/12 17:35:03
@@ -792,6 +792,30 @@
 return NULL;
 }
 
+/* Checks whether a filename is has a typical HTML suffix or not. The
+   following suffixes are presumed to be html files (case insensitive):
+   
+ html
+ htm
+ ?html (where ? is any character)
+
+   This is not necessarily a good indication that the file actually contains
+   HTML!  */
+int has_html_suffix_p (const char *fname)
+{
+  char *suf;
+
+  if ((suf = suffix (fname)) == NULL)
+return 0;
+  if (!strcasecmp (suf, "html"))
+return 1;
+  if (!strcasecmp (suf, "htm"))
+return 1;
+  if (suf[0] && !strcasecmp (suf + 1, "html"))
+return 1;
+  retu

Re: Referrer Faking and other nifty features

[Hrvoje Niksic]

"Ian Abbott" <[EMAIL PROTECTED]> writes:

> It would certainly make handling the logging output a bit of a
> challenge, especially the progress indication.

It would also require a completely different sort of organization, one
based on a central event loop.  There are programs that work that way,
such as `lftp', but Wget is not one of them and I don't think it will
become one any time soon.

I would much prefer to invest time into writing better http and ftp
backends, and supporting more protocols.

Re: /usr/include/stdio.h:120: previous declaration of `va_list'

[Kevin Rodgers]

Hrvoje Niksic writes:
>Kevin Rodgers <[EMAIL PROTECTED]> writes:
>
>> 1. Don't #define _XOPEN_SOURCE 500 (by commenting it out).
>>
>> 2. Do #define _VA_ALIST.
>>
>> I can confirm that (1) works.  I didn't try (2).
>
>Could you please try (2) and see if it works out?

OK, (2) also works (but it's _VA_LIST, not _VA_ALIST):

*** /usr/local/wget-1.8.1/src/config.h.orig Wed Mar  6 14:58:59 2002
--- /usr/local/wget-1.8.1/src/config.h  Fri Apr 12 11:46:42 2002
***
*** 263,268 
--- 263,269 
  /* For Solaris: request everything else that is available and doesn't
 conflict with the above.  */
  #define __EXTENSIONS__
+ #define _VA_LIST
  
  /* For Linux: request features of 4.3BSD and SVID (System V Interface
 Definition). */

>I'm reluctant to withdraw the _XOPEN_SOURCE definition because it's
>supposed to create the kind of "environment" that we want --
>standards-compliant with useful extensions.  Without it, some
>functions we use just don't get declared.  (I think strptime is one of
>them, but there are probably more.)  I'm keeping that option as a last
>resort.

I understand.

>Thanks for the report and the analysis.

You're welcome.  Thanks for responding.

-- 
Kevin

Re: Goodbye and good riddance

[Hrvoje Niksic]

"James C. McMaster (Jim)" <[EMAIL PROTECTED]> writes:

> This could be a great resource, but (I hate to say this) it has been
> rendered more trouble than it is worth by the stubbornness and
> stupidity of the owner.  He has turned a deaf ear to all pleas to do
> something, ANYTHING, to stop the flood of spam, viruses and
> annoyances posted to the list.

Actually, I was planning to work on the spam problem this weekend.
(Don't for a moment think I'm not annoyed by it.)  It *will* be
resolved, hopefully to everyone's satisfaction.  But if several spams
are enough to detract you from a "useful" resource and resort to
name-calling targeted at the very person who created it, I cannot
honestly feel dismayed by your choice.

> This is the one and only mailing list that still maintains this
> policy,

This is a factually incorrect statement.

> I will continue to use it without support, because getting support
> is more trouble than it is worth.

Don't forget that you can always post to the mailing list *without*
being subscribed.  :-)  Who knows, maybe one day you'll reap the
benefits of what you are badmouthing right now.


I respectfully ask the other participants to extend their patience for
some more days.  I apologize for not having provided a better solution
already.  Despite the insults, I do not deny my part of the blame --
it is just your method (of dealing with spam) I disagree with.

Re: /usr/include/stdio.h:120: previous declaration of `va_list'

[Charles Piety]

does anoyone know how i can unsubcsibe to all the wget mail lists?




|  Charles A. Piety|
|  Department of Meteorology   |
|  University of Maryland, College Park, MD 20742  |
|  phone(301) 405-7668 |
|  fax  (301) 314-9482  |
|  email: [EMAIL PROTECTED]|
|  homepage:  http://metosrv2.umd.edu/~charles/|
|  |



"Coming together is a beginning; Keeping together is progress;
 Working together is success."
--Anonymous

On Fri, 12 Apr 2002, Kevin Rodgers wrote:

> Hrvoje Niksic writes:
> >Kevin Rodgers <[EMAIL PROTECTED]> writes:
> >
> >> 1. Don't #define _XOPEN_SOURCE 500 (by commenting it out).
> >>
> >> 2. Do #define _VA_ALIST.
> >>
> >> I can confirm that (1) works.  I didn't try (2).
> >
> >Could you please try (2) and see if it works out?
>
> OK, (2) also works (but it's _VA_LIST, not _VA_ALIST):
>
> *** /usr/local/wget-1.8.1/src/config.h.orig   Wed Mar  6 14:58:59 2002
> --- /usr/local/wget-1.8.1/src/config.hFri Apr 12 11:46:42 2002
> ***
> *** 263,268 
> --- 263,269 
>   /* For Solaris: request everything else that is available and doesn't
>  conflict with the above.  */
>   #define __EXTENSIONS__
> + #define _VA_LIST
>
>   /* For Linux: request features of 4.3BSD and SVID (System V Interface
>  Definition). */
>
> >I'm reluctant to withdraw the _XOPEN_SOURCE definition because it's
> >supposed to create the kind of "environment" that we want --
> >standards-compliant with useful extensions.  Without it, some
> >functions we use just don't get declared.  (I think strptime is one of
> >them, but there are probably more.)  I'm keeping that option as a last
> >resort.
>
> I understand.
>
> >Thanks for the report and the analysis.
>
> You're welcome.  Thanks for responding.
>
> --
> Kevin
>
>
>

Re: Referrer Faking and other nifty features

[Thomas Lussnig]

>
>
>It would also require a completely different sort of organization, one
>based on a central event loop.  There are programs that work that way,
>such as `lftp', but Wget is not one of them and I don't think it will
>become one any time soon.
>
>I would much prefer to invest time into writing better http and ftp
>backends, and supporting more protocols.
>
There are 2 Protocols wich are very similar and one of them maybe 
interesting (NNTP and IMAP)
Because both of them could be represented in the same url style like 
http and ftp they also have
folders and "files".

What also could be nice is access to the p2p networks. But there is the 
"leech" problem. Because wget is only
intented as GET an not share.

Or what would beintersting protocols to support ?

Cu Thomas Lußnig



smime.p7s
Description: S/MIME Cryptographic Signature

unsubscribe [was: Re: /usr/include/stdio.h:120: previous declaration of `va_list']

[Noel Koethe]

On Fre, 12 Apr 2002, Charles Piety wrote:

> does anoyone know how i can unsubcsibe to all the wget mail lists?

see http://wget.sunsite.dk/ "Mailinglists"

-- 
Noèl Köthe

Re: Goodbye and good riddance

[James C. McMaster (Jim)]

In message <[EMAIL PROTECTED]>, Hrvoje Niksic said:
> "James C. McMaster (Jim)" <[EMAIL PROTECTED]> writes:
> 
> > This could be a great resource, but (I hate to say this) it has been
> > rendered more trouble than it is worth by the stubbornness and
> > stupidity of the owner.  He has turned a deaf ear to all pleas to do
> > something, ANYTHING, to stop the flood of spam, viruses and
> > annoyances posted to the list.
> 
> Actually, I was planning to work on the spam problem this weekend.
> (Don't for a moment think I'm not annoyed by it.)  It *will* be
> resolved, hopefully to everyone's satisfaction.  But if several spams
> are enough to detract you from a "useful" resource and resort to
> name-calling targeted at the very person who created it, I cannot
> honestly feel dismayed by your choice.
> 
"Several" does not begin to describe it.  This has been going on for a long 
time, and it is several per day.  This has been discussed for weeks, and you 
have never before expressed any inclination to "work on" the problem.  I do 
appreciate your effort in creating the list, but good intentions do not 
prevent a person's doing foolish things that lessen the good achieved by his 
efforts.

> > This is the one and only mailing list that still maintains this
> > policy,
> 
> This is a factually incorrect statement.
> 
Pardon me.  In my pique, I left out a phrase.  If should have been, "This is 
the one and only mailing list to which I subscribe that...".  My apologies.  
I tried to filter the spam by putting my wget-filing recipe after my 
spam-filtering recipes in .procmailrc, but for some reason that makes the 
wget-filing recipe fail intermittently.  I finally gave up after weeks of 
trying.  The material on this list is not worth the effort.
 
> > I will continue to use it without support, because getting support
> > is more trouble than it is worth.
> 
> Don't forget that you can always post to the mailing list *without*
> being subscribed.  :-)  Who knows, maybe one day you'll reap the
> benefits of what you are badmouthing right now.
> 
Not likely.  If I need to access the list, I will resubscribe.  If I want 
support, I will exert the miniscule effort to send an email to the right 
place.  I do not see why you feel that to be an onerous task.  If I want to 
be sure of seeing the answer, I will want to be on the list, because someone 
might answer on the list and not to me personally.  Then I will not see it if 
I am not subscribed.
> 
> I respectfully ask the other participants to extend their patience for
> some more days.  I apologize for not having provided a better solution
> already.  Despite the insults, I do not deny my part of the blame --
> it is just your method (of dealing with spam) I disagree with.
> 
My patience has reached an end.  Perhaps, now that you have (for the first 
time) indicated you will do something to fix the problem, the possible light 
at the end of the tunnel will convince others to stay.
-- 
Jim McMaster
mailto:[EMAIL PROTECTED]

Re: /usr/include/stdio.h:120: previous declaration of `va_list'

[Hrvoje Niksic]

Kevin Rodgers <[EMAIL PROTECTED]> writes:

>>Could you please try (2) and see if it works out?
>
> OK, (2) also works (but it's _VA_LIST, not _VA_ALIST):
[...]

Thanks.  This is the patch I'm about to apply:

2002-04-12  Hrvoje Niksic  <[EMAIL PROTECTED]>

* config.h.in: Define _VA_LIST on Solaris to prevent stdio.h from
declaring va_list.
From Kevin Rodgers <[EMAIL PROTECTED]>.

Index: src/config.h.in
===
RCS file: /pack/anoncvs/wget/src/config.h.in,v
retrieving revision 1.21
diff -u -r1.21 config.h.in
--- src/config.h.in 2002/04/12 01:14:18 1.21
+++ src/config.h.in 2002/04/12 18:33:48
@@ -257,6 +257,8 @@
 
 #ifdef solaris
 # define NAMESPACE_TWEAKS
+/* Prevent stdio.h from declaring va_list. */
+# define _VA_LIST
 #endif
 
 #ifdef __linux__

RESPONSE REQUIRED: Topica Email Verification

[Topica Support]

Hello [EMAIL PROTECTED],

We received your request to subscribe to the 
following email list(s), hosted free at Topica:

- FreebieList.com Free Stuff Newsletter


Click here to immediately activate your new
subscription(s):

http://www.topica.com/sysmsg/?cid=3.ebbbNTMTT.EEieebMG&p=ConfSub012

(Or cut and paste this link into your browser. If 
you prefer, you may also reply to this message.)


Please note: If you registered with Topica after
subscribing to the above list(s), your registration,
or any additional subscriptions you selected at
that time, will also be confirmed by clicking 
this link.


--
CONTACTING TOPICA

For questions about this subscription or your 
Topica account, please contact our Customer 
Support Department at: 
[EMAIL PROTECTED]

If you did not subscribe to the list(s) above, or 
feel we have sent this email in error, please 
forward this email to our abuse team for further 
review:
[EMAIL PROTECTED]

Sincerely,

Topica Customer Support

---
Topica - Learn More. Surf Less. 
The best place to find newsletters and discussions
on the topics you love.

Your Mailing List Subscription

[MailingList]

			
			Thank you for your mailing list subscriptions. If you have any questions about the content of any of the mailing you receive as a result of your subscription you may contact [EMAIL PROTECTED] 
			
			Anytime you would like to modify your subscription list you may do so by clicking on the link below.
			Then, enter your username and password as shown below:
			
			User Name: [EMAIL PROTECTED]
			Password: LKPEKFZ
			
			
			http://www.IgamingExchange.com/IGEXResearchTools.cfm?Action=ServicesDisplay&RToolType=NewsLetters 
			Your Current Email subscriptions include:
			
			

			IGamingNews Headlines	
			

			River City Group's Marketing Tools	
			

			River City Group's Match Play 	
			

			Supplier News 	
			
			
		
		

Your Mailing List Subscription

[MailingList]

		 	
			IGaming Exchange and IGaming News News Letter information
			You have chosen to remove yourself from all of the IGaming Exchange and
			IGaming News email list. If you have any questions or comments about the news
			letters please feel free to contact [EMAIL PROTECTED].
			Thank you,
			The River City Group Team
			
	
			 
		

Re: RESPONSE REQUIRED: Topica Email Verification

[Tony Lewis]

As if we don't get enough spam. Now someone is subscribing us to other
lists. 
- Original Message -
From: "Topica Support" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Friday, April 12, 2002 12:07 PM
Subject: RESPONSE REQUIRED: Topica Email Verification


> Hello [EMAIL PROTECTED],
>
> We received your request to subscribe to the
> following email list(s), hosted free at Topica:
>
> - FreebieList.com Free Stuff Newsletter
>
>
> Click here to immediately activate your new
> subscription(s):
>
> http://www.topica.com/sysmsg/?cid=3.ebbbNTMTT.EEieebMG&p=ConfSub012
>
> (Or cut and paste this link into your browser. If
> you prefer, you may also reply to this message.)
>
>
> Please note: If you registered with Topica after
> subscribing to the above list(s), your registration,
> or any additional subscriptions you selected at
> that time, will also be confirmed by clicking
> this link.
>
>
> --
> CONTACTING TOPICA
>
> For questions about this subscription or your
> Topica account, please contact our Customer
> Support Department at:
> [EMAIL PROTECTED]
>
> If you did not subscribe to the list(s) above, or
> feel we have sent this email in error, please
> forward this email to our abuse team for further
> review:
> [EMAIL PROTECTED]
>
> Sincerely,
>
> Topica Customer Support
>
> ---
> Topica - Learn More. Surf Less.
> The best place to find newsletters and discussions
> on the topics you love.
>

Re: Goodbye and good riddance

[Hrvoje Niksic]

"James C. McMaster (Jim)" <[EMAIL PROTECTED]> writes:

> "Several" does not begin to describe it.  This has been going on for
> a long time, and it is several per day.  This has been discussed for
> weeks, and you have never before expressed any inclination to "work
> on" the problem.

You are confusing my disagreement with your proposed solution with
disinclination to do anything.

> I do appreciate your effort in creating the list

When I said "creating the resource", I didn't mean the list, but Wget
itself.  The list was created and maintained by the good people at
sunsite.dk.

> Not likely.  If I need to access the list, I will resubscribe.  If I want 
> support, I will exert the miniscule effort to send an email to the right 
> place.  I do not see why you feel that to be an onerous task.

Because I don't want to be forced to subscribe to a mailing list to
get help.  Mailing lists are not newsgroups; they are considerably
harder to subscribe/unsubscribe, and the incoming traffic is generally
harder to deal with.

Re: Your Mailing List Subscription

[Ian Abbott]

On 12 Apr 2002 at 14:12, [EMAIL PROTECTED] wrote:

> IGaming Exchange and IGaming News News Letter information
> You have chosen to remove yourself from all of the IGaming Exchange 
> and IGaming News email list. If you have any questions or comments 
> about the news letters please feel free to contact 
> [EMAIL PROTECTED]
> Thank you,
> The River City Group Team

I'm not sure which helpful person subscribed [EMAIL PROTECTED] to
the above mailing lists in the first place, but hopefully I've done
the right thing by unsubscribing them again!

Re: Goodbye and good riddance

[sphargis]

   Give me a break.  just throw the unwanted mail
away like everyone else or set up a spam filter.

Big Deal..


--
Samuel Hargis
--
Systems Consultant/Software Developer
[EMAIL PROTECTED] - (850) 508-8583

Re: /usr/include/stdio.h:120: previous declaration of `va_list'

[Hrvoje Niksic]

I've now noticed that this solution fails under Sun's CC, because
va_list never gets defined.  Since this is needed only under gcc (to
prevent a clash between Sun's and gcc's va_list), I've conditionalized
the define under #ifdef __GNUC__.  I've tested the result on Solaris
8, and it seems to build on all compilers, without warnings.

This patch applies on top of the previous one:

2002-04-12  Hrvoje Niksic  <[EMAIL PROTECTED]>

* config.h.in: Only define _VA_LIST when compiled with gcc.

Index: src/config.h.in
===
RCS file: /pack/anoncvs/wget/src/config.h.in,v
retrieving revision 1.22
diff -u -r1.22 config.h.in
--- src/config.h.in 2002/04/12 18:36:07 1.22
+++ src/config.h.in 2002/04/12 20:25:06
@@ -257,8 +257,11 @@
 
 #ifdef solaris
 # define NAMESPACE_TWEAKS
-/* Prevent stdio.h from declaring va_list. */
-# define _VA_LIST
+# ifdef __GNUC__
+/* Prevent stdio.h from declaring va_list and thus tripping gcc's
+   stdarg.h. */
+#  define _VA_LIST
+# endif
 #endif
 
 #ifdef __linux__

Re: HTTP 1.1

[Hrvoje Niksic]

"Tony Lewis" <[EMAIL PROTECTED]> writes:

> Hrvoje Niksic wrote:
>
>> > Is there any way to make Wget use HTTP/1.1 ?
>>
>> Unfortunately, no.
>
> In looking at the debug output, it appears to me that wget is really
> sending HTTP/1.1 headers, but claiming that they are HTTP/1.0
> headers. For example, the Host header was not defined in RFC 1945,
> but wget is sending it.

Yes.  That is by design -- HTTP was meant to be extended in that way.
Wget is also requesting and accepting `Keep-Alive', using `Range', and
so on.

Csaba Raduly's patch would break Wget because it doesn't suppose the
"chunked" transfer-encoding.  Also, its understanding of persistent
connection might not be compliant with HTTP/1.1.

Goodbye and good riddance

[James C. McMaster (Jim)]

This could be a great resource, but (I hate to say this) it has been rendered 
more trouble than it is worth by the stubbornness and stupidity of the owner. 
 He has turned a deaf ear to all pleas to do something, ANYTHING, to stop the 
flood of spam, viruses and annoyances posted to the list.  This is the one 
and only mailing list that still maintains this policy, which worked in a 
now-passed, more innocent age.  Because of that policy, this is the only list 
which bombards me with junk email.

In order to preserve the ability of some hypothetical people who cannot be 
bothered to subscribe in order to post bugs, he is ruining this list for 
everyone who does have the interest/intelligence to subscribe.

wget is a great tool.  I will continue to use it without support, because 
getting support is more trouble than it is worth.

The "good riddance" is not aimed at the terrific people who volunteer their 
time to answer questions.  It is aimed at (1) the spammers who have ruined 
this list and (2) the idiot list owner who has let it happen and refuses to 
do anything about it.
-- 
Jim McMaster
mailto:[EMAIL PROTECTED]

F-R-E-E TRIAL

[ricci]

To opt out from future mailings
CLICK HERE

RE: HTTP 1.1

[Boaz Yahav]

So basically I only need to make this change and recompile?
I wish this was a switch :)

-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]
Sent: Friday, April 12, 2002 12:58 PM
Cc: Boaz Yahav; [EMAIL PROTECTED]
Subject: Re: HTTP 1.1



On 11/04/2002 18:26:15 hniksic wrote:

>"Boaz Yahav" <[EMAIL PROTECTED]> writes:
>
>> Is there any way to make Wget use HTTP/1.1 ?
>
>Unfortunately, no.

Sure it can be made to use HTTP 1.1

--- http.c.orig   Wed Jan 30 14:10:42 2002
+++ http.c  Fri Apr 12 11:56:22 2002
@@ -838,7 +838,7 @@
  + 64);
   /* Construct the request.  */
   sprintf (request, "\
-%s %s HTTP/1.0\r\n\
+%s %s HTTP/1.1\r\n\
 User-Agent: %s\r\n\
 Host: %s%s%s%s\r\n\
 Accept: %s\r\n\




:-)

--
Csaba Ráduly, Software Engineer   Sophos
Anti-Virus
email: [EMAIL PROTECTED]
http://www.sophos.com
US Support: +1 888 SOPHOS 9 UK Support: +44 1235
559933

Re: HTTP 1.1

[Tony Lewis]

Hrvoje Niksic wrote:

> > Is there any way to make Wget use HTTP/1.1 ?
>
> Unfortunately, no.

In looking at the debug output, it appears to me that wget is really sending
HTTP/1.1 headers, but claiming that they are HTTP/1.0 headers. For example,
the Host header was not defined in RFC 1945, but wget is sending it.

Tony

RE: HTTP 1.1

[Ian Abbott]

On 12 Apr 2002 at 21:41, Boaz Yahav wrote:

> So basically I only need to make this change and recompile?
> I wish this was a switch :)

No, it just lies to the server.  Wget is not really an HTTP/1.1
client and may go up in smoke if the server thinks it is!
(Hopefully it contains enough extinguisher to prevent a major fire,
but regardless, the result would not be as intended. ^_^)

Re: HTTP 1.1

[Ian Abbott]

On 12 Apr 2002 at 11:59, Tony Lewis wrote:

> Hrvoje Niksic wrote:
> 
> > > Is there any way to make Wget use HTTP/1.1 ?
> >
> > Unfortunately, no.
> 
> In looking at the debug output, it appears to me that wget is really sending
> HTTP/1.1 headers, but claiming that they are HTTP/1.0 headers. For example,
> the Host header was not defined in RFC 1945, but wget is sending it.

It's allowed to send these additional headers as a server would
just ignore those it doesn't understand.  Wget does not fulfill the
minimal requirements of an HTTP/1.1 client, which is why it does
not pretend to be one!  Maybe one day

Re: wget 1.6 inconveniences with FTP access through a FWTK firewall

[Hrvoje Niksic]

Paul Eggert <[EMAIL PROTECTED]> writes:

> I'm using wget 1.6 on Solaris 8 (sparc), and am connected to the
> Internet via a FWTK FTP proxy .
>
> If I want to retrieve a file via the standard Solaris 'ftp' command,
> without using 'wget', I do something like this:
>
>   $ ftp firewall
>   Connected to alioth.twinsun.com.
>   220 alioth FTP proxy (Version V2.1) ready.
>   Name (firewall:eggert): [EMAIL PROTECTED]
>   331-(GATEWAY CONNECTED TO elsie.nci.nih.gov)
>   331-(220 elsie.nci.nih.gov FTP server (Version wu-2.6.0(1) Thu Apr 27 22:04:37 
>EDT 2000) ready.)
>   331 Guest login ok, send your complete e-mail address as password.
>   Password:[EMAIL PROTECTED]>
>   230 Guest login ok, access restrictions apply.
[...]
> If I want to use wget to grab the same file, I have to do something
> like this:
>
>   $ wget ftp://anonymous%40elsie.nci.nih.gov@firewall/pub/pi.shar.gz

I've now implemented support for this.  You should be able to achieve
the same with:

$ export ftp_proxy=ftp://firewall
$ wget ftp://elsie.nci.nih.gov/pub/pi.shar.gz

The support is available in the latest CVS and is waiting for someone
to test it.  :-)