[Bug-wget] broken progressbar in 1.16

[Michael Shigorin]

Hello,
please do pay attention to regression testing for innovative
bells and especially whistles as the "new and improved" progress
bar quickly becomes annoying if the filename is longer than the
part of the line before percent meter: the new code tries to
scroll it (this alone might be irritating to some including
myself) and for some cases fails to do that properly falling into
"spam out a new line every cycle" in xterm with an UTF-8 locale.

e.g.,

$ LANG=ru_RU.UTF-8 wget -c 
http://fly.osdn.org.ua/~mike/iso/rescue/live-rescue-20141027-x86_64.iso
--2014-10-28 00:28:44--  
http://fly.osdn.org.ua/~mike/iso/rescue/live-rescue-20141027-x86_64.iso
Распознаётся fly.osdn.org.ua (fly.osdn.org.ua)… 212.40.45.6
Подключение к fly.osdn.org.ua (fly.osdn.org.ua)|212.40.45.6|:80... соединение 
установлено.
HTTP-запрос отправлен. Ожидание ответа... 200 OK
Длина: 620756992 (592M) [application/octet-stream]
Сохранение в: «live-rescue-20141027-x86_64.iso»

-rescue-20141027-x8   0%[  ]   1,25M  1,57MB/s  
rescue-20141027-x86   0%[  ]   1,65M  1,65MB/s  
escue-20141027-x86_   0%[  ]   2,06M  1,71MB/s  
scue-20141027-x86_6   0%[  ]   2,44M  1,74MB/s      
cue-20141027-x86_64   0%[  ]   2,84M  1,78MB/s  
ue-20141027-x86_64.   0%[  ]   3,26M  1,81MB/s  
e-20141027-x86_64.i   0%[  ]   3,68M  1,84MB/s  
 
^C

while disabling l18n would result in most of the spam missing:

$ LANG=C wget -c 
http://fly.osdn.org.ua/~mike/iso/rescue/live-rescue-20141027-x86_64.iso
converted 
'http://fly.osdn.org.ua/~mike/iso/rescue/live-rescue-20141027-x86_64.iso' 
(ANSI_X3.4-1968) -> 
'http://fly.osdn.org.ua/~mike/iso/rescue/live-rescue-20141027-x86_64.iso' 
(UTF-8)
--2014-10-28 00:30:09--  
http://fly.osdn.org.ua/~mike/iso/rescue/live-rescue-20141027-x86_64.iso
Resolving fly.osdn.org.ua (fly.osdn.org.ua)... 212.40.45.6
Connecting to fly.osdn.org.ua (fly.osdn.org.ua)|212.40.45.6|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 620756992 (592M) [application/octet-stream]
Saving to: 'live-rescue-20141027-x86_64.iso'

live-rescue-2014102   0%[  ]   4.55M  1.89MB/s 
^C

...but adding the "conversion" spam.

Guys, please do RCs or whatever to avoid brown paper bag effects
in products that were mature years ago.  I love you and thank for
your work but please please don't feel cool, people depend on wget.

-- 
  WBR, Michael Shigorin / http://altlinux.org
  -- http://opennet.ru / http://anna-news.info

Re: [Bug-wget] [PATCH] Small fix for limited number of strings (and potential garbage value) in arguments to concat_strings

[Pär Karlsson]

No complaints from me, it applied cleanly, all tests OK. :-)

2014-10-27 21:53 GMT+01:00 Tim Rühsen :

> Am Samstag, 25. Oktober 2014, 11:09:48 schrieb Pär Karlsson:
> > Good point. I got carried away by the criticism against xrealloc being
> too
> > expensive, etc, but you are of course right. Your implementation seems to
> > me the most readable, and in this context, it's what matters. :-)
> >
> > Thanks for the tip re: callgrind :-)
> >
> > /Pär
> >
> > 2014-10-24 20:39 GMT+02:00 Tim Rühsen :
> > > Pär, thanks for your work.
> > >
> > > *BUT* speed does not really matter here. My intention was to show code
> > > that is
> > > much more readable than the current implementation which is unnecessary
> > > complicated and not even understandable by the LLVM/clang analyzer.
> > > That piece of code should be replaced.
> > >
> > > FYI, if you want to work on CPU cycle optimization, use valgrind --
> > > tool=callgrind for your wget command to be polished. The resulting file
> > > can be
> > > viewed with e.g. kcachegrind.
> > >
> > > Tim
>
> i made up a patch.
>
> Any complaints ?
>
> Tim
>

Re: [Bug-wget] [PATCH] Add valgrind testing support via ./configure

[Tim Rühsen]

Am Samstag, 25. Oktober 2014, 23:40:41 schrieb Tim Rühsen:
> Am Sonntag, 19. Oktober 2014, 17:41:30 schrieb Darshit Shah:
> > On 10/09, Tim Rühsen wrote:
> > >> Hence, hard coding the command actually reduces the amount of work a
> > >> user
> > >> needs to do in order to run the tests under valgrind.
> > >>
> > >> My suggestion is that we allow the configure option, but hard code the
> > >> valgrind command into the test suites themselves, and not leave them
> > >> environment variables.
> > >
> > >What about removing the configure option and
> > >if VALGRIND_TESTS is undefined or empty or "0": normal testing
> > >if VALGRIND_TESTS is "1": valgrind testing with hard-coded options
> > >else: testing with command given in VALGRIND_TESTS
> > >
> > >The above described workflow should still work, right ?
> > >And we could do the complete test suite with
> > >VALGRIND_TESTS="1" make check
> > >
> > >What do you think ?
> >
> > --- end quoted text ---
> >
> > I actually like this idea.
> >
> > case $VALGRIND_TESTS:
> > "", 0) Normal tests;;
> > 1) Hard coded valgrind string;;
> > *) Execute the provided command;;
> >
> > Could you please make the relevant changes to atleast the Perl based tests
> > and to configure.ac? I'm currently traveling, but I'll fix the Python
> > tests
> > ASAP and send in a patch which will work well with the aforementioned
> > cases.
>
> Hi Darhsit,
>
> here is the valgrind patch. Both Perl and Python test suites amended.
>
> Tim

Added the ChangeLog entries.

Tim
From fc82fe93512e52b373800b1489efffb68b21b72e Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Tim Rühsen?= 
Date: Wed, 8 Oct 2014 11:03:45 +0200
Subject: [PATCH 2/2] add ./configure valgrind support to test suites

---
 ChangeLog |  4 
 configure.ac  | 20 
 testenv/ChangeLog |  6 ++
 testenv/Makefile.am   |  3 ++-
 testenv/README|  6 --
 testenv/test/base_test.py | 11 +++
 tests/ChangeLog   |  5 +
 tests/Makefile.am |  3 ++-
 tests/WgetTests.pm| 17 +
 9 files changed, 63 insertions(+), 12 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 51644a1..9998c7e 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,9 @@
 2014-10-25  Tim Ruehsen 

+	* configure.ac: add --enable-valgrind-tests
+
+2014-10-25  Tim Ruehsen 
+
 	* configure.ac: check for strlcpy()

 2014-10-22  Ángel González 
diff --git a/configure.ac b/configure.ac
index 56a4767..88401cf 100644
--- a/configure.ac
+++ b/configure.ac
@@ -101,6 +101,25 @@ test x"${ENABLE_DEBUG}" = xyes && AC_DEFINE([ENABLE_DEBUG], 1,
[Define if you want the debug output support compiled in.])

 dnl
+dnl Check for valgrind
+dnl
+AC_ARG_ENABLE(valgrind-tests,
+  AS_HELP_STRING([--enable-valgrind-tests], [enable using Valgrind for tests]),
+  [ac_enable_valgrind=$enableval], [ac_enable_valgrind=no])
+if test "${ac_enable_valgrind}" != "no" ; then
+  AC_CHECK_PROG(HAVE_VALGRIND, valgrind, yes, no)
+  if test "$HAVE_VALGRIND" = "yes" ; then
+VALGRIND_TESTS="1"
+AC_SUBST(VALGRIND_TESTS)
+VALGRIND_INFO="Test suite will be run under Valgrind"
+  else
+VALGRIND_INFO="Valgrind not found"
+  fi
+else
+  VALGRIND_INFO="Valgrind testing not enabled"
+fi
+
+dnl
 dnl Find the compiler
 dnl

@@ -599,4 +618,5 @@ AC_MSG_NOTICE([Summary of build options:
   NTLM:  $ENABLE_NTLM
   OPIE:  $ENABLE_OPIE
   Debugging: $ENABLE_DEBUG
+  Valgrind:  $VALGRIND_INFO
 ])
diff --git a/testenv/ChangeLog b/testenv/ChangeLog
index 18087b6..c57a431 100644
--- a/testenv/ChangeLog
+++ b/testenv/ChangeLog
@@ -1,3 +1,9 @@
+2014-10-25  Tim Ruehsen 
+
+	* test/base_test.py (gen_cmd_line): generate valgrind command line if requested
+	* README: amend description of VALGRIND_TESTS
+	* Makefile.am: set/export VALGRIND_TESTS
+
 2014-10-01  Darshit Shah  

 	* Makefile.am: Run the tests in Python's Optimizedmode
diff --git a/testenv/Makefile.am b/testenv/Makefile.am
index b1f6781..33604bc 100644
--- a/testenv/Makefile.am
+++ b/testenv/Makefile.am
@@ -27,7 +27,8 @@


 AUTOMAKE_OPTIONS = parallel-tests
-AM_TESTS_ENVIRONMENT = export WGETRC=/dev/null; MAKE_CHECK=True; export MAKE_CHECK; export PYTHONPATH=$$PYTHONPATH:$(srcdir);
+AM_TESTS_ENVIRONMENT = export WGETRC=/dev/null; MAKE_CHECK=True; export MAKE_CHECK;\
+ export PYTHONPATH=$$PYTHONPATH:$(srcdir); export VALGRIND_TESTS="@VALGRIND_TESTS@";
 TESTS = Test-auth-basic-fail.py \
 Test-auth-basic.py  \
 Test-auth-both.py   \
diff --git a/testenv/README b/testenv/README
index 413e12e..081a957 100644
--- a/testenv/README
+++ b/testenv/README
@@ -93,8 +93,10 @@ Environment Variables:
   valgrind.
 * NO_CLEANUP: Do not remove the temporary files created by the test.
   This will prevent the ${testname}-test directory from being deleted
-* VALGRIND_TESTS: If this variable is set, the test suite will execute all the
-  tests through val

Re: [Bug-wget] [PATCH] Small fix for limited number of strings (and potential garbage value) in arguments to concat_strings

[Tim Rühsen]

Am Samstag, 25. Oktober 2014, 11:09:48 schrieb Pär Karlsson:
> Good point. I got carried away by the criticism against xrealloc being too
> expensive, etc, but you are of course right. Your implementation seems to
> me the most readable, and in this context, it's what matters. :-)
>
> Thanks for the tip re: callgrind :-)
>
> /Pär
>
> 2014-10-24 20:39 GMT+02:00 Tim Rühsen :
> > Pär, thanks for your work.
> >
> > *BUT* speed does not really matter here. My intention was to show code
> > that is
> > much more readable than the current implementation which is unnecessary
> > complicated and not even understandable by the LLVM/clang analyzer.
> > That piece of code should be replaced.
> >
> > FYI, if you want to work on CPU cycle optimization, use valgrind --
> > toolÊllgrind for your wget command to be polished. The resulting file
> > can be
> > viewed with e.g. kcachegrind.
> >
> > Tim

i made up a patch.

Any complaints ?

Tim
From cd7ed2dc2251913fc55398de2057713f2ba1e1e6 Mon Sep 17 00:00:00 2001
From: Tim Ruehsen 
Date: Sat, 25 Oct 2014 21:03:05 +0200
Subject: [PATCH 1/2] added strlcpy(), concat_strings() rewritten

Signed-off-by: Tim Ruehsen 
---
 ChangeLog |  6 -
 configure.ac  |  2 +-
 src/ChangeLog |  6 +
 src/utils.c   | 71 +--
 src/utils.h   |  4 
 5 files changed, 56 insertions(+), 33 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index eca59da..51644a1 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,4 +1,8 @@
-2013-10-22  Ángel González 
+2014-10-25  Tim Ruehsen 
+
+	* configure.ac: check for strlcpy()
+
+2014-10-22  Ángel González 

 	* bootstrap.conf (gnulib_modules): Add module xstrndup.

diff --git a/configure.ac b/configure.ac
index 3cbe618..56a4767 100644
--- a/configure.ac
+++ b/configure.ac
@@ -207,7 +207,7 @@ AC_FUNC_MMAP
 AC_FUNC_FSEEKO
 AC_CHECK_FUNCS(strptime timegm vsnprintf vasprintf drand48 pathconf)
 AC_CHECK_FUNCS(strtoll usleep ftello sigblock sigsetjmp memrchr wcwidth mbtowc)
-AC_CHECK_FUNCS(sleep symlink utime)
+AC_CHECK_FUNCS(sleep symlink utime strlcpy)

 if test x"$ENABLE_OPIE" = xyes; then
   AC_LIBOBJ([ftp-opie])
diff --git a/src/ChangeLog b/src/ChangeLog
index d0f753f..c58a475 100644
--- a/src/ChangeLog
+++ b/src/ChangeLog
@@ -1,3 +1,9 @@
+<<< HEAD
+2014-10-25  Tim Ruehsen  
+
+	* utils.c: added strlcpy(), concat_strings() rewritten
+	* utils.h: added strlcpy()
+
 2014-09-08  Darshit Shah  

 	* ftp.c (ftp_retrieve_glob): Also check for invalid entries along with
diff --git a/src/utils.c b/src/utils.c
index 78c282e..3280294 100644
--- a/src/utils.c
+++ b/src/utils.c
@@ -349,6 +349,32 @@ aprintf (const char *fmt, ...)
 #endif /* not HAVE_VASPRINTF */
 }

+#ifndef HAVE_STRLCPY
+/* strlcpy() is a BSD function that sometimes is really handy.
+ * It is the same as snprintf(dst,dstsize,"%s",src), but much faster. */
+
+size_t
+strlcpy (char *dst, const char *src, size_t size)
+{
+  const char *old = src;
+
+  /* Copy as many bytes as will fit */
+  if (size)
+{
+  while (--size)
+{
+  if (!(*dst++ = *src++))
+return src - old - 1;
+}
+
+  *dst = 0;
+}
+
+  while (*src++);
+  return src - old - 1;
+}
+#endif
+
 /* Concatenate the NULL-terminated list of string arguments into
freshly allocated space.  */

@@ -356,47 +382,30 @@ char *
 concat_strings (const char *str0, ...)
 {
   va_list args;
-  int saved_lengths[5]; /* inspired by Apache's apr_pstrcat */
-  char *ret, *p;
-
-  const char *next_str;
-  int total_length = 0;
-  size_t argcount;
+  const char *arg;
+  size_t length = 0, pos = 0;
+  char *s;

-  /* Calculate the length of and allocate the resulting string. */
+  if (!str0)
+return NULL;

-  argcount = 0;
+  /* calculate the length of the resulting string */
   va_start (args, str0);
-  for (next_str = str0; next_str != NULL; next_str = va_arg (args, char *))
-{
-  int len = strlen (next_str);
-  if (argcount < countof (saved_lengths))
-saved_lengths[argcount++] = len;
-  total_length += len;
-}
+  for (arg = str0; arg; arg = va_arg (args, const char *))
+length += strlen(arg);
   va_end (args);
-  p = ret = xmalloc (total_length + 1);

-  /* Copy the strings into the allocated space. */
+  s = xmalloc (length + 1);

-  argcount = 0;
+  /* concatenate strings */
   va_start (args, str0);
-  for (next_str = str0; next_str != NULL; next_str = va_arg (args, char *))
-{
-  int len;
-  if (argcount < countof (saved_lengths))
-len = saved_lengths[argcount++];
-  else
-len = strlen (next_str);
-  memcpy (p, next_str, len);
-  p += len;
-}
+  for (arg = str0; arg; arg = va_arg (args, const char *))
+pos += strlcpy(s + pos, arg, length - pos + 1);
   va_end (args);
-  *p = '\0';

-  return ret;
+  return s;
 }
-
+
 /* Format the provided time according to the specified format.  The
format is a string with format elements supported by strftime.  */

diff -

[Bug-wget] [PATCH 3/5] use pkg-config to check pcre deps if available

[Mike Frysinger]

Newer versions of these packages ship with pkg-config files, so if we can
detect it via those, do so.  If that fails, fall back to the old methods.

Also add a configure flag to explicitly control its usage.
---
 configure.ac | 25 ++---
 1 file changed, 18 insertions(+), 7 deletions(-)

diff --git a/configure.ac b/configure.ac
index a5fb67d..8d2050e 100644
--- a/configure.ac
+++ b/configure.ac
@@ -581,13 +581,24 @@ dnl
 dnl Check for PCRE
 dnl
 
-AC_CHECK_HEADER(pcre.h,
-AC_CHECK_LIB(pcre, pcre_compile,
-  [LIBS="${LIBS} -lpcre"
-   AC_DEFINE([HAVE_LIBPCRE], 1,
- [Define if libpcre is available.])
-  ])
-)
+AC_ARG_ENABLE(pcre, AC_HELP_STRING([--disable-pcre],
+   [Disable PCRE style regular expressions]))
+
+AS_IF([test "X$enable_pcre" != "Xno"],[
+  PKG_CHECK_MODULES([PCRE], libpcre, [
+LIBS="$PCRE_LIBS $LIBS"
+CFLAGS="$PCRE_CFLAGS $CFLAGS"
+AC_DEFINE([HAVE_LIBPCRE], [1], [Define if using libpcre.])
+  ], [
+AC_CHECK_HEADER(pcre.h,
+AC_CHECK_LIB(pcre, pcre_compile,
+  [LIBS="${LIBS} -lpcre"
+   AC_DEFINE([HAVE_LIBPCRE], 1,
+ [Define if libpcre is available.])
+  ])
+)
+  ])
+])
 
 
 dnl Needed by src/Makefile.am
-- 
2.1.2

[Bug-wget] [PATCH] openssl: fix implicit decl warning

[Mike Frysinger]

Include the right header to fix:

openssl.c: In function 'ssl_init':
openssl.c:195:3: warning: implicit declaration of function
'ENGINE_load_builtin_engines' [-Wimplicit-function-declaration]
   ENGINE_load_builtin_engines();
   ^
---
 src/openssl.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/src/openssl.c b/src/openssl.c
index e24954a..2a3e0d6 100644
--- a/src/openssl.c
+++ b/src/openssl.c
@@ -42,6 +42,7 @@ as that of the covered work.  */
 #include 
 #if OPENSSL_VERSION_NUMBER >= 0x00907000
 #include 
+#include 
 #endif
 
 #include "utils.h"
-- 
2.1.2

[Bug-wget] [PATCH 1/5] use pkg-config to check zlib deps if available

[Mike Frysinger]

Newer versions of these packages ship with pkg-config files, so if we can
detect it via those, do so.  If that fails, fall back to the old methods.
---
 configure.ac | 10 +-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/configure.ac b/configure.ac
index 3cbe618..8ec6c09 100644
--- a/configure.ac
+++ b/configure.ac
@@ -244,9 +244,17 @@ dnl
 dnl Checks for libraries.
 dnl
 
+PKG_PROG_PKG_CONFIG
+
 AS_IF([test x"$with_zlib" != xno], [
   with_zlib=yes
-  AC_CHECK_LIB(z, compress)
+  PKG_CHECK_MODULES([ZLIB], zlib, [
+LIBS="$ZLIB_LIBS $LIBS"
+CFLAGS="$ZLIB_CFLAGS $CFLAGS"
+AC_DEFINE([HAVE_LIBZ], [1], [Define if using zlib.])
+  ], [
+AC_CHECK_LIB(z, compress)
+  ])
 ])
 
 AS_IF([test x"$with_ssl" = xopenssl], [
-- 
2.1.2

[Bug-wget] [PATCH 2/5] use pkg-config to check libuuid deps if available

[Mike Frysinger]

Newer versions of these packages ship with pkg-config files, so if we can
detect it via those, do so.  If that fails, fall back to the old methods.

Also add a configure flag to explicitly control its usage.
---
 configure.ac | 38 +-
 1 file changed, 25 insertions(+), 13 deletions(-)

diff --git a/configure.ac b/configure.ac
index 8ec6c09..a5fb67d 100644
--- a/configure.ac
+++ b/configure.ac
@@ -550,20 +550,32 @@ fi
 dnl
 dnl Check for UUID
 dnl
-AC_CHECK_HEADER(uuid.h,
-AC_CHECK_FUNC(uuid, uuid_create,
-  [AC_DEFINE([HAVE_UUID_CREATE], 1,
- [Define if uuid_create is available.])
-  ])
-)
 
-AC_CHECK_HEADER(uuid/uuid.h,
-AC_CHECK_LIB(uuid, uuid_generate,
-  [LIBS="${LIBS} -luuid"
-   AC_DEFINE([HAVE_LIBUUID], 1,
- [Define if libuuid is available.])
-  ])
-)
+AC_ARG_WITH(libuuid, AC_HELP_STRING([--without-libuuid],
+[Generate UUIDs for WARC files via 
libuuid]))
+
+AS_IF([test "X$with_libuuid" != "Xno"],[
+  PKG_CHECK_MODULES([UUID], uuid, [
+LIBS="$UUID_LIBS $LIBS"
+CFLAGS="$UUID_CFLAGS $CFLAGS"
+AC_DEFINE([HAVE_LIBUUID], [1], [Define if using libuuid.])
+  ], [
+AC_CHECK_HEADER(uuid.h,
+AC_CHECK_FUNC(uuid, uuid_create,
+  [AC_DEFINE([HAVE_UUID_CREATE], 1,
+ [Define if uuid_create is available.])
+  ])
+)
+
+AC_CHECK_HEADER(uuid/uuid.h,
+AC_CHECK_LIB(uuid, uuid_generate,
+  [LIBS="${LIBS} -luuid"
+   AC_DEFINE([HAVE_LIBUUID], 1,
+ [Define if libuuid is available.])
+  ])
+)
+  ])
+])
 
 dnl
 dnl Check for PCRE
-- 
2.1.2

[Bug-wget] [PATCH 5/5] use pkg-config to check gnutls deps if available

[Mike Frysinger]

Newer versions of these packages ship with pkg-config files, so if we can
detect it via those, do so.  If that fails, fall back to the old methods.
---
 configure.ac | 28 ++--
 1 file changed, 18 insertions(+), 10 deletions(-)

diff --git a/configure.ac b/configure.ac
index ec55fbb..3ca39be 100644
--- a/configure.ac
+++ b/configure.ac
@@ -329,18 +329,26 @@ AS_IF([test x"$with_ssl" = xopenssl], [
 with_ssl=gnutls
 
 dnl Now actually check for -lgnutls
-AC_LIB_HAVE_LINKFLAGS([gnutls], [], [
-#include 
-], [gnutls_global_init()])
-if test x"$LIBGNUTLS" != x
-then
-  ssl_found=yes
+PKG_CHECK_MODULES([GNUTLS], [gnutls], [
   AC_MSG_NOTICE([compiling in support for SSL via GnuTLS])
   AC_LIBOBJ([gnutls])
-  LIBS="$LIBGNUTLS $LIBS"
-else
-  AC_MSG_ERROR([--with-ssl=gnutls was given, but GNUTLS is not available.])
-fi
+  LIBS="$GNUTLS_LIBS $LIBS"
+  CFLAGS="$GNUTLS_CFLAGS -DHAVE_LIBGNUTLS $CFLAGS"
+  AC_DEFINE([HAVE_LIBGNUTLS], [1], [Define if using gnutls.])
+], [
+  AC_LIB_HAVE_LINKFLAGS([gnutls], [], [
+#include 
+  ], [gnutls_global_init()])
+  if test x"$LIBGNUTLS" != x
+  then
+ssl_found=yes
+AC_MSG_NOTICE([compiling in support for SSL via GnuTLS])
+AC_LIBOBJ([gnutls])
+LIBS="$LIBGNUTLS $LIBS"
+  else
+AC_MSG_ERROR([--with-ssl=gnutls was given, but GNUTLS is not 
available.])
+  fi
+])
 
 AC_CHECK_FUNCS(gnutls_priority_set_direct)
   ]) # endif: --with-ssl != no?
-- 
2.1.2

[Bug-wget] [PATCH 4/5] use pkg-config to check openssl deps if available

[Mike Frysinger]

Newer versions of these packages ship with pkg-config files, so if we can
detect it via those, do so.  If that fails, fall back to the old methods.
---
 configure.ac | 100 ---
 1 file changed, 54 insertions(+), 46 deletions(-)

diff --git a/configure.ac b/configure.ac
index 8d2050e..ec55fbb 100644
--- a/configure.ac
+++ b/configure.ac
@@ -258,42 +258,50 @@ AS_IF([test x"$with_zlib" != xno], [
 ])
 
 AS_IF([test x"$with_ssl" = xopenssl], [
-  dnl As of this writing (OpenSSL 0.9.6), the libcrypto shared library
-  dnl doesn't record its dependency on libdl, so we need to make sure
-  dnl -ldl ends up in LIBS on systems that have it.  Most OSes use
-  dnl dlopen(), but HP-UX uses shl_load().
-  AC_CHECK_LIB(dl, dlopen, [], [
-AC_CHECK_LIB(dl, shl_load)
-  ])
-
-  ssl_found=no
-  case $host_os in
-*mingw32* )
-  dnl prefer link to openssl dlls if possible. if not then fallback on 
static libs. if not then error
+  PKG_CHECK_MODULES([OPENSSL], [openssl], [
+AC_MSG_NOTICE([compiling in support for SSL via OpenSSL])
+AC_LIBOBJ([openssl])
+LIBS="$OPENSSL_LIBS $LIBS"
+CFLAGS="$OPENSSL_CFLAGS -DHAVE_LIBSSL $CFLAGS"
+LIBSSL=" " # ntlm check below wants this
+AC_DEFINE([HAVE_LIBSSL], [1], [Define if using openssl.])
+  ], [
+dnl As of this writing (OpenSSL 0.9.6), the libcrypto shared library
+dnl doesn't record its dependency on libdl, so we need to make sure
+dnl -ldl ends up in LIBS on systems that have it.  Most OSes use
+dnl dlopen(), but HP-UX uses shl_load().
+AC_CHECK_LIB(dl, dlopen, [], [
+  AC_CHECK_LIB(dl, shl_load)
+])
 
-  AC_CHECK_LIB(eay32, EVP_MD_CTX_init)
-  if test x"$ac_cv_lib_eay32_EVP_MD_CTX_init" != xno
-  then
-AC_CHECK_LIB(ssl32, SSL_connect, [
-  ssl_found=yes
-  AC_MSG_NOTICE([Enabling support for SSL via OpenSSL (shared)])
-  AC_LIBOBJ([openssl])
-  LIBS="${LIBS} -lssl32"
-  AC_DEFINE([HAVE_LIBSSL32], [1], [Define to 1 if you have the `ssl32' 
library (-lssl32).])
-],
-AC_MSG_ERROR([openssl not found: shared lib eay32 found but ssl32 not 
found]))
-
-  else
-LIBS+=' -lgdi32'
-dnl fallback and test static libs
-  fi
-  dnl add zdll lib as dep for above tests?
-;;
-  esac
+ssl_found=no
+case $host_os in
+  *mingw32* )
+dnl prefer link to openssl dlls if possible. if not then fallback on 
static libs. if not then error
+
+AC_CHECK_LIB(eay32, EVP_MD_CTX_init)
+if test x"$ac_cv_lib_eay32_EVP_MD_CTX_init" != xno
+then
+  AC_CHECK_LIB(ssl32, SSL_connect, [
+ssl_found=yes
+AC_MSG_NOTICE([Enabling support for SSL via OpenSSL (shared)])
+AC_LIBOBJ([openssl])
+LIBS="${LIBS} -lssl32"
+AC_DEFINE([HAVE_LIBSSL32], [1], [Define to 1 if you have the 
`ssl32' library (-lssl32).])
+  ],
+  AC_MSG_ERROR([openssl not found: shared lib eay32 found but ssl32 
not found]))
+
+else
+  LIBS+=' -lgdi32'
+  dnl fallback and test static libs
+fi
+dnl add zdll lib as dep for above tests?
+  ;;
+esac
 
-  AS_IF([test x$ssl_found != xyes], [
-dnl Now actually check for -lssl if it wasn't already found
-AC_LIB_HAVE_LINKFLAGS([ssl], [crypto], [
+AS_IF([test x$ssl_found != xyes], [
+  dnl Now actually check for -lssl if it wasn't already found
+  AC_LIB_HAVE_LINKFLAGS([ssl], [crypto], [
 #include 
 #include 
 #include 
@@ -301,19 +309,19 @@ AS_IF([test x"$with_ssl" = xopenssl], [
 #include 
 #include 
 #include 
-], [SSL_library_init ()])
-if test x"$LIBSSL" != x
-then
-  ssl_found=yes
-  AC_MSG_NOTICE([compiling in support for SSL via OpenSSL])
-  AC_LIBOBJ([openssl])
-  LIBS="$LIBSSL $LIBS"
-elif test x"$with_ssl" != x
-then
-  AC_MSG_ERROR([--with-ssl=openssl was given, but SSL is not available.])
-fi
+  ], [SSL_library_init ()])
+  if test x"$LIBSSL" != x
+  then
+ssl_found=yes
+AC_MSG_NOTICE([compiling in support for SSL via OpenSSL])
+AC_LIBOBJ([openssl])
+LIBS="$LIBSSL $LIBS"
+  elif test x"$with_ssl" != x
+  then
+AC_MSG_ERROR([--with-ssl=openssl was given, but SSL is not available.])
+  fi
+])
   ])
-
 ], [
   # --with-ssl is not openssl: check if it's no
   AS_IF([test x"$with_ssl" != xno], [
-- 
2.1.2

[Bug-wget] [PATCH] tests: fix skip exit code

[Mike Frysinger]

The test harness looks for exit code 77 to mark tests as skipped, not
exit 2.  Switch over so we get SKIP instead of FAIL.
---
 tests/WgetFeature.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/WgetFeature.pm b/tests/WgetFeature.pm
index 0762314..118e79c 100644
--- a/tests/WgetFeature.pm
+++ b/tests/WgetFeature.pm
@@ -22,7 +22,7 @@ sub import
 
 unless ($have_features{$feature}) {
 print $skip_messages{$feature}, "\n";
-exit 2; # skip
+exit 77; # skip
 }
 }
 
-- 
2.1.2

Re: [Bug-wget] COMMENT_IF_NO_POD2MAN broken?

[Dan McDonald]

> On Oct 27, 2014, at 11:23 AM, Dan McDonald  wrote:



> Has anyone tested this on a box without POD2MAN?  NOTE: if you use OmniOS or 
> some other Illumos distro, there's an additional patch needed for one source 
> file for 1.16, but that's an Illumos issue.

I've worked around this bug for the OmniOS build system, but (unlike the 
additional patch), not handling a missing pod2man binary seems to be a real 
problem.  I will gladly test a proposed fix out for anyone who's interested.

Thanks,
Dan

[Bug-wget] COMMENT_IF_NO_POD2MAN broken?

[Dan McDonald]

Hello from OmniOS (an illumos distro).  Giuseppe told me to send this report 
here.

Consider this diff:

   
http://git.savannah.gnu.org/cgit/wget.git/commit/doc/Makefile.am?id=d1ab00cab4d644d1048ed279f90e633b82cf4455

I've just updated wget (today's CVE release), but can't compile it thanks to 
gmake not coping with the lack of POD2MAN.  The 1.15 method(s) worked, but the 
new 1.16 method(s) does(do) not:

osdev3(wget-1.16/doc)[0]% gmake
./texi2pod.pl -D VERSION="1.16" ./wget.texi wget.pod
no --center="GNU Wget" --release="GNU Wget 1.16" wget.pod > wget.1
/bin/sh: line 1: no: not found
gmake: *** [wget.1] Error 127
osdev3(wget-1.16/doc)[2]% gmake --version
GNU Make 3.82
Built for i386-pc-solaris2.11
Copyright (C) 2010  Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
osdev3(wget-1.16/doc)[0]% 

Has anyone tested this on a box without POD2MAN?  NOTE: if you use OmniOS or 
some other Illumos distro, there's an additional patch needed for one source 
file for 1.16, but that's an Illumos issue.

Thanks,
Dan McDonald -- OmniOS engineering

Sent from my iPhone (typos, autocorrect, and all)

Re: [Bug-wget] Regression in git master branch (commit 8e6de1fb5ff0ca0c749da7db634a1b1e3a1215a2)

[Giuseppe Scrivano]

Tim Ruehsen  writes:

> On Thursday 23 October 2014 08:40:34 Giuseppe Scrivano wrote:
>> Ángel González  writes:
>> > Sigh, I had locally added xstrndup to bootstrap.conf but failed to
>> > include it in the commit.
>> > I'm fine with the changes. Feel free to push.
>> 
>> pushed!
>> 
>> Thanks,
>> Giuseppe
>
> Hi Giuseppe,
>
> I just made a git pull of the 1.16 version.
> And it seems that this patch is missing... thus CSS parsing is broken.
>
> (BTW, same as https://savannah.gnu.org/bugs/?42953)
>
> How could this happen ?

was the patch pushed?  We have only fast-forward push enabled for the
repository so history is linear and we can't force any push.

Anyway, we were in a hurry for this release to fix the reported security
issue, we can make a new one soon.

Regards,
Giuseppe

Re: [Bug-wget] Regression in git master branch (commit 8e6de1fb5ff0ca0c749da7db634a1b1e3a1215a2)

[Tim Ruehsen]

On Monday 27 October 2014 15:16:08 Tim Ruehsen wrote:
> On Thursday 23 October 2014 08:40:34 Giuseppe Scrivano wrote:
> > Ángel González  writes:
> > > Sigh, I had locally added xstrndup to bootstrap.conf but failed to
> > > include it in the commit.
> > > I'm fine with the changes. Feel free to push.
> > 
> > pushed!
> > 
> > Thanks,
> > Giuseppe
> 
> Hi Giuseppe,
> 
> I just made a git pull of the 1.16 version.
> And it seems that this patch is missing... thus CSS parsing is broken.
> 
> (BTW, same as https://savannah.gnu.org/bugs/?42953)
> 
> How could this happen ?
> 
> Tim

Arg, sorry for the noise.

It was in a not-up-to-date local branch.
master looks fine !!!

Tim

Re: [Bug-wget] Regression in git master branch (commit 8e6de1fb5ff0ca0c749da7db634a1b1e3a1215a2)

[Tim Ruehsen]

On Thursday 23 October 2014 08:40:34 Giuseppe Scrivano wrote:
> Ángel González  writes:
> > Sigh, I had locally added xstrndup to bootstrap.conf but failed to
> > include it in the commit.
> > I'm fine with the changes. Feel free to push.
> 
> pushed!
> 
> Thanks,
> Giuseppe

Hi Giuseppe,

I just made a git pull of the 1.16 version.
And it seems that this patch is missing... thus CSS parsing is broken.

(BTW, same as https://savannah.gnu.org/bugs/?42953)

How could this happen ?

Tim

Re: [Bug-wget] [Bug-Wget] Patch Test-proxied-https-auth.px

[Tim Ruehsen]

On Monday 27 October 2014 12:09:45 Tim Ruehsen wrote:
> On Monday 27 October 2014 11:51:24 Tim Ruehsen wrote:
> > On Sunday 26 October 2014 21:29:14 Darshit Shah wrote:
> > > Ever since the Perl tests have been ported to the parallel test harness,
> > > Test-proxied-https-auth.px has always failed for me. In a prior thread,
> > > we'd identified the problem as a race between the HTTPS server being
> > > set-up and Wget sending the request.
> > >
> > > The test was already using `sleep 1` to eliminate this race. I've bumped
> > > this up to 2 seconds. Now on my machine, the test no longer fails.
> > >
> > > If everyone is okay with this change, lets push it. If someone wants to
> > > change the number, please speak up.
> >
> > Hi Darshit,
> >
> > I made many test runs in the last time and never ever had
> > Test-proxied-https-auth.px fail here nor an a second machine (at home).
> >
> > AFAIR, we did not identify a race condition (or let's say a design flaw).
> > The test suite starts the server and talks with it before starting Wget.
> >
> > Please append your patch - I would like to further investigate what is
> > really going wrong (first of all I would like to reproduce the failure). I
> > assume the sleep just disguises a deeper problem that may pop up later in
> > one or another configuration.
>
> Now i am answering my own mail ;-)
>
> Darshit is right. Test-proxied-https-auth.px brings it's own server code
> that does not wait / sync parent and child. Thus we have a a design flaw
> here... which already has been 'fixed' in WgetTests.pm. Just let me take a
> look if I can transfer some of the code to avoid the race without using
> sleep.
>
> Or: move Test-proxied-https-auth to the Python test suite !?

Darshit, could you please test this version of Test-proxied-https-auth.px ?

It adds a sync between parent and child, so a sleep isn't needed.

Tim


Test-proxied-https-auth.px
Description: Perl program


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] [Bug-Wget] Patch Test-proxied-https-auth.px

[Tim Ruehsen]

On Monday 27 October 2014 11:51:24 Tim Ruehsen wrote:
> On Sunday 26 October 2014 21:29:14 Darshit Shah wrote:
> > Ever since the Perl tests have been ported to the parallel test harness,
> > Test-proxied-https-auth.px has always failed for me. In a prior thread,
> > we'd identified the problem as a race between the HTTPS server being
> > set-up and Wget sending the request.
> > 
> > The test was already using `sleep 1` to eliminate this race. I've bumped
> > this up to 2 seconds. Now on my machine, the test no longer fails.
> > 
> > If everyone is okay with this change, lets push it. If someone wants to
> > change the number, please speak up.
> 
> Hi Darshit,
> 
> I made many test runs in the last time and never ever had
> Test-proxied-https-auth.px fail here nor an a second machine (at home).
> 
> AFAIR, we did not identify a race condition (or let's say a design flaw).
> The test suite starts the server and talks with it before starting Wget.
> 
> Please append your patch - I would like to further investigate what is
> really going wrong (first of all I would like to reproduce the failure). I
> assume the sleep just disguises a deeper problem that may pop up later in
> one or another configuration.

Now i am answering my own mail ;-)

Darshit is right. Test-proxied-https-auth.px brings it's own server code that 
does not wait / sync parent and child. Thus we have a a design flaw here... 
which already has been 'fixed' in WgetTests.pm. Just let me take a look if I 
can transfer some of the code to avoid the race without using sleep.

Or: move Test-proxied-https-auth to the Python test suite !?

Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] [Bug-Wget] Patch Test-proxied-https-auth.px

[Tim Ruehsen]

On Sunday 26 October 2014 21:29:14 Darshit Shah wrote:
> Ever since the Perl tests have been ported to the parallel test harness,
> Test-proxied-https-auth.px has always failed for me. In a prior thread, we'd
> identified the problem as a race between the HTTPS server being set-up and
> Wget sending the request.
> 
> The test was already using `sleep 1` to eliminate this race. I've bumped
> this up to 2 seconds. Now on my machine, the test no longer fails.
> 
> If everyone is okay with this change, lets push it. If someone wants to
> change the number, please speak up.

Hi Darshit,

I made many test runs in the last time and never ever had 
Test-proxied-https-auth.px fail here nor an a second machine (at home).

AFAIR, we did not identify a race condition (or let's say a design flaw). The 
test suite starts the server and talks with it before starting Wget.

Please append your patch - I would like to further investigate what is really 
going wrong (first of all I would like to reproduce the failure). I assume the 
sleep just disguises a deeper problem that may pop up later in one or another 
configuration.

Tim


signature.asc
Description: This is a digitally signed message part.

[Bug-wget] GNU wget 1.16 released

[Giuseppe Scrivano]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hello,

I am pleased to announce the new version of GNU wget.

It is available for download here:

ftp://ftp.gnu.org/gnu/wget/wget-1.16.tar.gz
ftp://ftp.gnu.org/gnu/wget/wget-1.16.tar.xz

and the GPG detached signatures using the key E163E1EA:

ftp://ftp.gnu.org/gnu/wget/wget-1.16.tar.gz.sig
ftp://ftp.gnu.org/gnu/wget/wget-1.16.tar.xz.sig

To reduce load on the main server, you can use this redirector service
which automatically redirects you to a mirror:

http://ftpmirror.gnu.org/wget/wget-1.16.tar.gz
http://ftpmirror.gnu.org/wget/wget-1.16.tar.xz

* Noteworthy changes in Wget 1.16

** No longer create local symbolic links by default.  Closes CVE-2014-4877.

** Use libpsl for verifying cookie domains.

** Default progress bar output changed.

** Introduce --show-progress to force display the progress bar.

** Introduce --no-config.  The wgetrc files will not be read.

** Introduce --start-pos to allow starting downloads from a specified position.

** Fix a problem with ISA Server Proxy and keep-alive connections.

Many thanks to anyone who contributed to this release!

Please report any problem you may experience to the bug-wget@gnu.org
mailing list.

Have fun!
Giuseppe
-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: Processed by Mailcrypt 3.5.9 

iQIcBAEBAgAGBQJUThmrAAoJECY9bfLhY+Hqtz8P/RkO9GP8g8wiQqeaGVbJ6oNi
N4AnG/W1E0OfTDSFNG8Bd4sqA6a32gInIPkz/gFsl4l52MRABcFKXm64yLPB9Xmj
vtuZKmca6nAlfwb5SYmapVfeXKTXIcotBGuOEWruM69dD5C2kdAzO7Epv1Rtf5Cy
I89HHib4sEpQ2Mnwijx5OtwTdzBNTzF0S2lAIOQGztCqcViMLxJAIJ3+JCd1LNZ6
V0rte70zrIH0eYeZsxRcL61uGef7XB5kVjZcyE1J6/QJDkWdVWTFeGikOUI+wCFA
pTiddTZpcRuoVzwKlvqRCloXAQ1bPDFE5Sw+8+0E3dkhKXkGuswaXMyCV1Z5Xjy3
opXYQ827B2D3znGnPWpHFokJO16rE6p8jwkPC+CKOThY38lCl0l++rcmLqKFg+PA
HTP0DwKB1tWU/n8c2EkwFiCWQaxqU3lt4Hpn+H9NsWOTdeG7cfM/1+cjLxCOoP7X
hRzFOVkR7Ket7ytF0MGjFLwBlCqzjo7yt7+2R2E2zqme78svsZLyNdgyYHwOJjbW
oKIQHp3DY3TjdKt4uRRb49NZqT9VwsF6snpqRFveNGJGyRshs1RWSlKG9P7TIULw
65b8ZzH7F6c/SuMk11GMBML+8BWRsK7IY6251ZMOFTXjfLy5ZhCb85uFaZvMf9ud
OHxpd6zFvXsp3M8mYd6i
=RUfr
-END PGP SIGNATURE-

Re: [Bug-wget] [Win32 Patch] console-close events

[Tim Ruehsen]

On Tuesday 21 October 2014 12:46:29 Gisle Vanem wrote:
> Darit, about the git format-patch. I don't know how.
> 
> The src/Changelog entry could simply be:

After you 'git clone' you make your changes (ideally in your own branch).

You commit your changes to your local repo
git commit -a -m "description"

Now you generate a patchfile with
git format-patch -1
and send the resulting file to this list.
(Others would use  'git am < file.patch' to apply your patch.)

Here is what I do when someone asks me to amend a patch:
1. make the changes
2. commit them (git commit -a m "blabla")
3. merge the last two commits into one with interactive 'git rebase':
git rebase -i HEAD~2
Your editor will open with two lines beginning with 'pick'.
Change 'pick' of line with 'blabla' to 'f' and save the file.

Of course there are (quite a few) other ways to achieve the same (e.g. you 
could undo the last commit with 'git reset').

If you are creating an own branch for each patch your are working on, you 
should be pretty clean. You can always checkout master and make a 'git pull' 
to get the latest changes from upstream.

You will find good help and many examples (for special problems one might 
have) in the internet.

Tim


signature.asc
Description: This is a digitally signed message part.