Re: wget 1.10 problems under AIX
Hi Hrvoje,
Thanks for the detailed report!
Thanks for your detailed answer ;-)
Jens Schleusener [EMAIL PROTECTED] writes:
1) Only using the configure-option --disable-nls and the C compiler
gcc 4.0.0 the wget-binary builds successfully
I'd be interested in seeing the error log without --disable-nls and/or
with the system compiler.
I will send that logs to your personal mail-address.
although gcc outputs
some compiler warnings like
convert.c: In function 'convert_all_links':
convert.c:95: warning: incompatible implicit declaration of built-in
function 'alloca'
The alloca-declaring magic in config-post.h (taken from the Autoconf
manual) apparently doesn't take into account that GCC wants alloca
declared too.
But simply calling the new wget like
wget http://www.example.com/
I got always errors like
--12:36:51-- http://www.example.com/
= `index.html'
Resolving www.example.com... failed: Invalid flags in hints.
This is really bad. Apparently your version of getaddrinfo is broken
or Wget is using it incorrectly. Can you intuit which flags cause the
problem? Depending on the circumstances, Wget uses AI_ADDRCONFIG,
AI_PASSIVE, and/or AI_NUMERICHOST.
Yes, all three seems defined, probably via /usr/include/netdb.h. Here an
extract of that file:
/* Flag definitions for addrinfo hints in protocol-independent
name/addr/service service. RFC2133 */
/* Also flag definitions for getipnodebyname RFC 2553 */
#define AI_CANONNAME0x01/* canonical name to be included in return */
#define AI_PASSIVE 0x02/* prepare return for call to bind() */
#define AI_NUMERICHOST 0x04/* RFC 2553, nodename is a numeric host
address string */
#define AI_ADDRCONFIG 0x08/* RFC 2553, source address family configured */
#define AI_V4MAPPED 0x10/* RFC 2553, accept v4 mapped addresses */
#define AI_ALL 0x20/* RFC 2553, accept all addresses */
#define AI_DEFAULT (AI_V4MAPPED | AI_ADDRCONFIG) /* RFC 2553 */
But I have no idea were the error message Invalid flags in hints comes
from. Directly from wget (probably not) or from system resolver routines?
After some testing I found that using the additional
configure-option --disable-ipv6 solves that problem.
Because it disables IPv6 (and therefore the use of getaddrinfo)
altogether.
Ok, that was not clear to me.
2) Using the additional configure-option --with-ssl=/usr/local/contrib
fails although the openssl (0.9.7g) header files are installed under
/usr/local/contrib/include/openssl/ and the libssl.a under
/usr/local/contrib/lib/.
This is not a standard layout, so the configure script is having
problems with it. The supported layouts are one of:
* No flags are needed, the includes are found without -Iincludedir,
and the library gets linked in without the need for -Llibdir.
* The library is installed in $root, which means that includes are in
$root/include and the libraries in $root/lib. OpenSSL's own default
for $root is /usr/local/ssl, which Wget checks for.
To resolve situations like this, Wget should probably support
specifying additional include and library directories separately.
I believe you can work around this by specifying:
./configure CPPFLAGS=-I/usr/local/contrib/include -L/usr/local/contrib/lib
Can you check if that works for you?
That doesn't works (typo ?) better seems
./configure CPPFLAGS=-I/usr/local/contrib/include
LDFLAGS=-L/usr/local/contrib/lib
but that also doesn't solve the described problem.
Also the configure option
--with-ssl[=SSL-ROOT]
respectively in my case
--with-ssl=/usr/local/contrib
should probably do that job.
After long trial and error testing I have the impression that the
configure-script has an error. If I change for e.g. at line 25771
{ ac_try='test -s conftest.$ac_objext'
into
{ ac_try='test -s .libs/conftest.$ac_objext'
the generated test object file will now be found. Therefore also my
openssl-installation will be found and compiled successfully into wget.
3) Using the native IBM C compiler (CC=cc) instead GNU gcc I got the
compile error
cc -qlanglvl=ansi -I. -I. -I/opt/include -DHAVE_CONFIG_H
-DSYSTEM_WGETRC=\/usr/local/contrib/etc/wgetrc\
-DLOCALEDIR=\/usr/local/contrib/share/locale\ -O -c main.c
main.c, line 147.16: 1506-275 (S) Unexpected text ',' encountered.
Simply changing line 147 of src/main.c from
OPT__PARENT,
to
OPT__PARENT
let the compile error vanish (sorry, I am not a C expert).
[...]
It's a newer C feature that leaked into Wget -- sorry about that.
I'll try to get these fixed in Wget 1.10.1.
Your patch works.
Greetings
Jens
--
Dr. Jens SchleusenerT-Systems Solutions for Research GmbH
Tel: +49 551 709-2493 Bunsenstr.10
Fax: +49 551 709-2169 D-37073 Goettingen
[EMAIL PROTECTED] http://www.t-systems.com/
Re: wget 1.10 problems under AIX
Hi Hrvoje,
Jens Schleusener [EMAIL PROTECTED] writes:
--12:36:51-- http://www.example.com/
= `index.html'
Resolving www.example.com... failed: Invalid flags in hints.
This is really bad. Apparently your version of getaddrinfo is broken
or Wget is using it incorrectly. Can you intuit which flags cause the
problem? Depending on the circumstances, Wget uses AI_ADDRCONFIG,
AI_PASSIVE, and/or AI_NUMERICHOST.
Yes, all three seems defined, probably via /usr/include/netdb.h.
Then I am guessing that AIX's getaddrinfo doesn't like AF_UNSPEC
family + AI_ADDRCONFIG hint. If you use `wget -4
http://www.example.com/', does it then work?
Works.
But I have no idea were the error message Invalid flags in hints
comes from. Directly from wget (probably not) or from system
resolver routines?
From the system resolver, which Wget invokes via getaddrinfo.
That doesn't works (typo ?) better seems
./configure CPPFLAGS=-I/usr/local/contrib/include
LDFLAGS=-L/usr/local/contrib/lib
That's what I meant, sorry. But that is pretty much what
--with-ssl=/usr/local/include does. (I misread your original message,
thinking that the OpenSSL includes were in an entirely different
location).
respectively in my case
--with-ssl=/usr/local/contrib
should probably do that job.
Yes. I'd like to see config.log, or the relevant parts thereof, which
should contain errors.
Here the config.log extract:
configure:25735: looking for SSL libraries in /usr/local/contrib
configure:25742: checking for includes
configure:25756: /bin/sh ./libtool gcc -c -O2 -Wall -Wno-implicit
-I/usr/local/contrib/include conftest.c 5
gcc -c -O2 -Wall -Wno-implicit -I/usr/local/contrib/include conftest.c
-DPIC -o .libs/conftest.o
*** Warning: inferring the mode of operation is deprecated.
*** Future versions of Libtool will require --mode=MODE be specified.
configure:25762: $? = 0
configure:25766: test -z
|| test ! -s conftest.err
configure:25769: $? = 0
configure:25772: test -s conftest.o
configure:25775: $? = 1
configure: failed program was:
|
| #include openssl/ssl.h
| #include openssl/x509.h
| #include openssl/err.h
| #include openssl/rand.h
| #include openssl/des.h
| #include openssl/md4.h
| #include openssl/md5.h
|
configure:25787: result: not found
configure:26031: error: failed to find OpenSSL libraries
The reason for the above error is as already written - at least in my case
using the self compiled libtool version 1.5 - that the configure script
tests for the non-existing conftest.o instead for the generated and
existing .libs/conftest.o.
The above line
configure:25756: /bin/sh ./libtool gcc -c -O2 -Wall -Wno-implicit
-I/usr/local/contrib/include conftest.c 5
gcc -c -O2 -Wall -Wno-implicit -I/usr/local/contrib/include conftest.c
-DPIC -o .libs/conftest.o
looks for me (as a configure-layman) a little bit strange (gcc twice?).
Here an corresponding extract of config.log (from the same system) while
configuring lynx2.8.6dev.13 where I have no such problems (but the
configure script and the conftest.c file looks different):
configure:8128: checking for openssl include directory
configure:8145: gcc -c -I/usr/local/contrib/include
-I/usr/local/contrib/include -D_ACS_COMPAT_CODE -D_POSIX_C_SOURCE=199506L
conftest.c 5
configure:8148: $? = 0
configure:8151: test -s conftest.o
configure:8154: $? = 0
configure:8163: result: yes
After long trial and error testing I have the impression that the
configure-script has an error. If I change for e.g. at line 25771
{ ac_try='test -s conftest.$ac_objext'
into
{ ac_try='test -s .libs/conftest.$ac_objext'
the generated test object file will now be found.
But why don't I (and other non-AIX testers) have that problem? Maybe
Libtool is doing something strange on AIX?
I will try to re-compile current used libtool version 1.5 under AIX 5.1
(may be it was built under AIX 4.3) and compile and use the newest libtool
(version 1.5.18).
Greetings
Jens
--
Dr. Jens SchleusenerT-Systems Solutions for Research GmbH
Tel: +49 551 709-2493 Bunsenstr.10
Fax: +49 551 709-2169 D-37073 Goettingen
[EMAIL PROTECTED] http://www.t-systems.com/
Re: wget 1.10 problems under AIX
Hi,
The above line
configure:25756: /bin/sh ./libtool gcc -c -O2 -Wall -Wno-implicit
-I/usr/local/contrib/include conftest.c 5
gcc -c -O2 -Wall -Wno-implicit -I/usr/local/contrib/include conftest.c
-DPIC -o .libs/conftest.o
looks for me (as a configure-layman) a little bit strange (gcc
twice?).
GCC is only invoked twice. The first line is configure telling you
the command it is about run. The second line is libtool telling you
exactly how it is about to run GCC.
For me there is no `-o .libs/conftest.o', even though I use the same
libtool invocation on my system.
Sorry, if I bother you but here I see a difference under AIX 5.1 to for
e.g. my SuSE 9.3 system:
AIX 5.1 (SSL-ROOT=/usr/local/contrib):
==
configure:25742: checking for includes
configure:25756: /bin/sh ./libtool gcc -c -O2 -Wall -Wno-implicit
-I/usr/local/contrib/include conftest.c 5
gcc -c -O2 -Wall -Wno-implicit -I/usr/local/contrib/include conftest.c -DPIC
-o .libs/conftest.o
*** Warning: inferring the mode of operation is deprecated.
*** Future versions of Libtool will require --mode=MODE be specified.
configure:25762: $? = 0
configure:25766: test -z
|| test ! -s conftest.err
configure:25769: $? = 0
configure:25772: test -s conftest.o
configure:25775: $? = 1
SuSE 9.3 (SSL-ROOT=/usr):
=A
configure:25742: checking for includes
configure:25756: /bin/sh ./libtool gcc -c -O2 -Wall -Wno-implicit
-I/usr/include conftest.c 5
gcc -c -O2 -Wall -Wno-implicit -I/usr/include conftest.c -fPIC -DPIC
-o .libs/conftest.o
gcc -c -O2 -Wall -Wno-implicit -I/usr/include conftest.c
-o conftest.o /dev/null 21
*** Warning: inferring the mode of operation is deprecated.
*** Future versions of Libtool will require --mode=MODE be specified.
configure:25762: $? = 0
configure:25766: test -z
|| test ! -s conftest.err
configure:25769: $? = 0
configure:25772: test -s conftest.o
configure:25775: $? = 0
configure:25778: result: found
That output is produces from the configure line 25756:
if { (eval echo $as_me:$LINENO: \$ac_compile\) 5
The real compiling is done by the next line 25756:
(eval $ac_compile) 2conftest.er1
The content of $ac_compile seems to be (SuSE)
/bin/sh ./libtool gcc -c -O2 -Wall -Wno-implicit -I/usr/include conftest.c 5
but that produces under SuSE (Linux) obviously two gcc processes and two
object files (.libs/conftest.o AND conftest.o) so that the objectfile
existing test (searching only for conftest.o) is successful. But under AIX
only one object file (.libs/conftest.o) is generated that the object file
existing test doesn't find.
Greetings
Jens
--
Dr. Jens SchleusenerT-Systems Solutions for Research GmbH
Tel: +49 551 709-2493 Bunsenstr.10
Fax: +49 551 709-2169 D-37073 Goettingen
[EMAIL PROTECTED] http://www.t-systems.com/
wget 1.10 problems under AIX
Hi, under AIX 5.1 I have some problems with the new wget 1.10 release. 1) Only using the configure-option --disable-nls and the C compiler gcc 4.0.0 the wget-binary builds successfully although gcc outputs some compiler warnings like convert.c: In function 'convert_all_links': convert.c:95: warning: incompatible implicit declaration of built-in function 'alloca' and cookies.c:644: warning: format '%d' expects type 'int', but argument 4 has type 'long int' But simply calling the new wget like wget http://www.example.com/ I got always errors like --12:36:51-- http://www.example.com/ = `index.html' Resolving www.example.com... failed: Invalid flags in hints. After some testing I found that using the additional configure-option --disable-ipv6 solves that problem. 2) Using the additional configure-option --with-ssl=/usr/local/contrib fails although the openssl (0.9.7g) header files are installed under /usr/local/contrib/include/openssl/ and the libssl.a under /usr/local/contrib/lib/. Extract of config.log: configure:25735: looking for SSL libraries in /usr/local/contrib configure:25742: checking for includes configure:25756: /bin/sh ./libtool gcc -c -O2 -Wall -Wno-implicit -I/usr/local/ contrib/include conftest.c 5 gcc -c -O2 -Wall -Wno-implicit -I/usr/local/contrib/include conftest.c -DPIC - o .libs/conftest.o *** Warning: inferring the mode of operation is deprecated. *** Future versions of Libtool will require --mode=MODE be specified. configure:25762: $? = 0 configure:25766: test -z || test ! -s conftest.err configure:25769: $? = 0 configure:25772: test -s conftest.o configure:25775: $? = 1 configure: failed program was: | | #include openssl/ssl.h | #include openssl/x509.h | #include openssl/err.h | #include openssl/rand.h | #include openssl/des.h | #include openssl/md4.h | #include openssl/md5.h | configure:25787: result: not found configure:26031: error: failed to find OpenSSL libraries 3) Using the native IBM C compiler (CC=cc) instead GNU gcc I got the compile error cc -qlanglvl=ansi -I. -I. -I/opt/include -DHAVE_CONFIG_H -DSYSTEM_WGETRC=\/usr/local/contrib/etc/wgetrc\ -DLOCALEDIR=\/usr/local/contrib/share/locale\ -O -c main.c main.c, line 147.16: 1506-275 (S) Unexpected text ',' encountered. Simply changing line 147 of src/main.c from OPT__PARENT, to OPT__PARENT let the compile error vanish (sorry, I am not a C expert). But using the required configure-option --disable-ipv6 I got then the similar error cc -qlanglvl=ansi -I. -I. -I/opt/include -DHAVE_CONFIG_H -DSYSTEM_WGETRC=\/usr/local/contrib/etc/wgetrc\ -DLOCALEDIR=\/usr/local/contrib/share/locale\ -O -c connect.c host.h, line 52.17: 1506-275 (S) Unexpected text ',' encountered. Again removing the , on line 52 helps and I got a working wget binary. Greetings Jens P.S.: Under Linux SuSE 9.3 all compiles and works well. -- Dr. Jens SchleusenerT-Systems Solutions for Research GmbH Tel: +49 551 709-2493 Bunsenstr.10 Fax: +49 551 709-2169 D-37073 Goettingen [EMAIL PROTECTED] http://www.t-systems.com/
How to use wget with option -p without writing files to disk?
Hi, I just want to use wget (v1.9.1-rc1) to do some simple access-time benchmarking of some WWW pages. So I first started with wget --page-requisites --timeout=30 --proxy=off \ --tries=1 \ http://www.foo.bar/ (last output line for e.g.: Downloaded: 76,431 bytes in 27 files) But then I remarked that in this way I also measured the disk I/O while writing the fetched files to the local disk. So the next idea was to let write wget the output to /dev/null (option --tries=1 omitted since it's the default using --output-document) wget --page-requisites --timeout=30 --proxy=off \ --output-document=/dev/null \ http://www.foo.bar/ (last output line for e.g.: Downloaded: 31,999 bytes in 2 files) But that doesn't work since wget probably needs the downloaded pages to find the files necessary to properly display the complete HTML page. A workaround seems to call wget once in the standard way so the files are locally available but that probably wouldn't work correctly if the benchmarked page were be changed. Any ideas to that correctly with wget? Or any pointers to more appropriate tools? Greetings Jens -- Dr. Jens SchleusenerT-Systems Solutions for Research GmbH Tel: +49 551 709-2493 Bunsenstr.10 Fax: +49 551 709-2169 D-37073 Goettingen [EMAIL PROTECTED] http://www.t-systems.com/
