Re: [pacman-dev] Download options

2016-12-02 Thread Allan McRae 
On 03/12/16 06:31, Olivier Brunel wrote:
> On Fri, 2 Dec 2016 16:47:41 +1000
> Allan McRae  wrote:
> 
>> There was many thread and patches regarding this, so I am starting
>> another!
>>
>> So far there are two (maybe three) things people want options for to
>> do with the downloader.  I see most of these options being set in
>> pacman.conf and rarely specified on the command line.  Argue with me
>> if I am wrong here...
>>
>> Based on that assumption, here are my preferred option names (for both
>> pacman.conf and the command line):
>>
>> 1) Disable low speed timeout
>>
>> DisableDownloadTimeout
>> --disable-download-timeout
>>
>> I see no need to separate out speed and time timeouts.
>>
>>
>> 2) Set maximum download speed
>>
>> MaxDownloadSpeed
>> --max-download-speed
>>
>> I would also accept replacing speed with limit.
>>
>>
>> 3) (in a glorious future) Set maximum concurrent downloads
>>
>> MaxConcurrentDownloads
>> --max-concurrent-downloads
>>
>>
>> Any argument about these option names will need to be very clearly
>> justified.  I don't own a bike so have no need for a bikeshed.
> 
> Well, I could re-state how "Dl" is shorter than "Download" which might
> be nice, especially on command-line option -- but also looks to me like
> all those options would be about the longest ones yet, so far options
> can have "shortcuts" in their names, e.g. --dbpath not --databasepath or
> --nodeps not --nodependencies
> 
> It is possible those might be rarely specified on command line, doesn't
> mean they have to be as lengthy as possible because of it. Clear but
> short is good, and I feel that can be achieved here.
> 
> Doesn't have to apply to options in pacman.conf, and I guess they don't
> have to be named the same; There's already difference in case style, and
> it wouldn't even be the first time, e.g. --ignore vs IgnorePkg
> 
> 
> Why use "disable" as prefix while every other options usually goes with
> "no" instead? (--nodeps, --noconfirm, --noscriplet, etc)
> 
> 
> And while on the topic of consistency, let me add that almost all of
> pacman's options use a "single word style," i.e. no dashes in them:
> --ignoregroup; --noprogressbar; --downloadonly; etc Why not keep that
> up?
> 
> Yes, there are already expections to the rule (namely --print-format &
> --assume-installed), but those are the only two (I believe), I'm not
> sure it's good to make more. Consistency is a nice thing.
> 
> 
> So at least for command line, --nodltimeout and --maxdlspeed (and for
> the glorious future simply --maxdownloads) seem better to me.
> (Though if you don't like the "dl" bit, --nodownloadtimeout &
> --maxdownloadspeed would work too I guess; I do prefer the
> shorter versions myself, which I find still pretty clear/readable,
> but your choice.)
> 
> Either way we can use NoDownloadTimeout & MaxDownloadSpeed (and
> MawDownloads, or even MaxConcurrentDownloads) in pacman.conf; Yes the
> "No" prefix is also used there, so again feels better/more consistent
> that way.
> 

And we hit bikeshed again...

I'm done.  All patches adding new options will not be considered until
further notice.

Allan

Re: [pacman-dev] [PATCH 1/1] add generated scripts to gitignore

2016-12-02 Thread Allan McRae 
On 02/12/16 20:58, Christian Hesse wrote:
> From: Christian Hesse 
> 
> These files are generated at build time. Ignore in git.
> 

Thanks!

Re: [pacman-dev] Download options

2016-12-02 Thread Allan McRae 
On 03/12/16 06:54, Dave Reisner wrote:
> On Fri, Dec 02, 2016 at 04:47:41PM +1000, Allan McRae wrote:
>> There was many thread and patches regarding this, so I am starting another!
>>
>> So far there are two (maybe three) things people want options for to do
>> with the downloader.  I see most of these options being set in
>> pacman.conf and rarely specified on the command line.  Argue with me if
>> I am wrong here...
>>
>> Based on that assumption, here are my preferred option names (for both
>> pacman.conf and the command line):
>>
>> 1) Disable low speed timeout
>>
>> DisableDownloadTimeout
>> --disable-download-timeout
> 
> As a user, I would have no idea what this means. As the person who wrote
> the code that this affects, the thing which is most closely associated
> with time in this equation can hardly be called a "timeout". It's more
> of a threshold.
> 
> Some other suggestions, because my shed is better than your shed:
> 
> --disabledeadnetworkcheck
> --nocheckdeadnetwork
> --nodeadconncheck
> --nodeadconnectioncheck
> --namingishard
> 

I think timeout captures what this is trying to do and is the commonly
used term for this kind of functionality in various downloaders.  What
it actually captures (10sec at less than 1kb/s) can be documented in the
man page.

>> I see no need to separate out speed and time timeouts.
>>
>>
>> 2) Set maximum download speed
>>
>> MaxDownloadSpeed
>> --max-download-speed
> 
> I think I've pointed out before that this is a lie at best. You aren't
> actually throttling anything, you're just adding artificial delay. You
> can also do this from outside of pacman:
> 
> https://lists.archlinux.org/pipermail/pacman-dev/2016-September/021367.html
> 
> FWIW, it's no longer true that curl uses a flat average -- it's now a
> rolling window which should provide a smoother limiting mechanism.
> However, that's a very recent change (7.50.2), and requires a libcurl
> far newer than what we need for compilation (7.32.0).
> 

I realise this.  It is the intended effect of the option that I am
naming.  The fact that curl does not implement it very well should be
documented in the man page for that option.

And despite our assertion that pacman is cross distro, almost every
usage of it will have recent curl...

>> I would also accept replacing speed with limit.
>>
>>
>> 3) (in a glorious future) Set maximum concurrent downloads
>>
>> MaxConcurrentDownloads
>> --max-concurrent-downloads
>>
>>
>> Any argument about these option names will need to be very clearly
>> justified.  I don't own a bike so have no need for a bikeshed.
>>
>> Allan
> .
>

Re: [pacman-dev] Download options

2016-12-02 Thread Dave Reisner 
On Fri, Dec 02, 2016 at 04:47:41PM +1000, Allan McRae wrote:
> There was many thread and patches regarding this, so I am starting another!
> 
> So far there are two (maybe three) things people want options for to do
> with the downloader.  I see most of these options being set in
> pacman.conf and rarely specified on the command line.  Argue with me if
> I am wrong here...
> 
> Based on that assumption, here are my preferred option names (for both
> pacman.conf and the command line):
> 
> 1) Disable low speed timeout
> 
> DisableDownloadTimeout
> --disable-download-timeout

As a user, I would have no idea what this means. As the person who wrote
the code that this affects, the thing which is most closely associated
with time in this equation can hardly be called a "timeout". It's more
of a threshold.

Some other suggestions, because my shed is better than your shed:

--disabledeadnetworkcheck
--nocheckdeadnetwork
--nodeadconncheck
--nodeadconnectioncheck
--namingishard

> I see no need to separate out speed and time timeouts.
> 
> 
> 2) Set maximum download speed
> 
> MaxDownloadSpeed
> --max-download-speed

I think I've pointed out before that this is a lie at best. You aren't
actually throttling anything, you're just adding artificial delay. You
can also do this from outside of pacman:

https://lists.archlinux.org/pipermail/pacman-dev/2016-September/021367.html

FWIW, it's no longer true that curl uses a flat average -- it's now a
rolling window which should provide a smoother limiting mechanism.
However, that's a very recent change (7.50.2), and requires a libcurl
far newer than what we need for compilation (7.32.0).

> I would also accept replacing speed with limit.
> 
> 
> 3) (in a glorious future) Set maximum concurrent downloads
> 
> MaxConcurrentDownloads
> --max-concurrent-downloads
> 
> 
> Any argument about these option names will need to be very clearly
> justified.  I don't own a bike so have no need for a bikeshed.
> 
> Allan

Re: [pacman-dev] Download options

2016-12-02 Thread Olivier Brunel 
On Fri, 2 Dec 2016 16:47:41 +1000
Allan McRae  wrote:

> There was many thread and patches regarding this, so I am starting
> another!
> 
> So far there are two (maybe three) things people want options for to
> do with the downloader.  I see most of these options being set in
> pacman.conf and rarely specified on the command line.  Argue with me
> if I am wrong here...
> 
> Based on that assumption, here are my preferred option names (for both
> pacman.conf and the command line):
> 
> 1) Disable low speed timeout
> 
> DisableDownloadTimeout
> --disable-download-timeout
> 
> I see no need to separate out speed and time timeouts.
> 
> 
> 2) Set maximum download speed
> 
> MaxDownloadSpeed
> --max-download-speed
> 
> I would also accept replacing speed with limit.
> 
> 
> 3) (in a glorious future) Set maximum concurrent downloads
> 
> MaxConcurrentDownloads
> --max-concurrent-downloads
> 
> 
> Any argument about these option names will need to be very clearly
> justified.  I don't own a bike so have no need for a bikeshed.

Well, I could re-state how "Dl" is shorter than "Download" which might
be nice, especially on command-line option -- but also looks to me like
all those options would be about the longest ones yet, so far options
can have "shortcuts" in their names, e.g. --dbpath not --databasepath or
--nodeps not --nodependencies

It is possible those might be rarely specified on command line, doesn't
mean they have to be as lengthy as possible because of it. Clear but
short is good, and I feel that can be achieved here.

Doesn't have to apply to options in pacman.conf, and I guess they don't
have to be named the same; There's already difference in case style, and
it wouldn't even be the first time, e.g. --ignore vs IgnorePkg


Why use "disable" as prefix while every other options usually goes with
"no" instead? (--nodeps, --noconfirm, --noscriplet, etc)


And while on the topic of consistency, let me add that almost all of
pacman's options use a "single word style," i.e. no dashes in them:
--ignoregroup; --noprogressbar; --downloadonly; etc Why not keep that
up?

Yes, there are already expections to the rule (namely --print-format &
--assume-installed), but those are the only two (I believe), I'm not
sure it's good to make more. Consistency is a nice thing.


So at least for command line, --nodltimeout and --maxdlspeed (and for
the glorious future simply --maxdownloads) seem better to me.
(Though if you don't like the "dl" bit, --nodownloadtimeout &
--maxdownloadspeed would work too I guess; I do prefer the
shorter versions myself, which I find still pretty clear/readable,
but your choice.)

Either way we can use NoDownloadTimeout & MaxDownloadSpeed (and
MawDownloads, or even MaxConcurrentDownloads) in pacman.conf; Yes the
"No" prefix is also used there, so again feels better/more consistent
that way.

Hopefully there were some clear justifications in there. :)

Cheers,
-j

[pacman-dev] [PATCH 1/1] add generated scripts to gitignore

2016-12-02 Thread Christian Hesse 
From: Christian Hesse 

These files are generated at build time. Ignore in git.

Signed-off-by: Christian Hesse 
---
 scripts/libmakepkg/.gitignore | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/scripts/libmakepkg/.gitignore b/scripts/libmakepkg/.gitignore
index 941d39a..ceb7bfc 100644
--- a/scripts/libmakepkg/.gitignore
+++ b/scripts/libmakepkg/.gitignore
@@ -10,5 +10,8 @@ srcinfo.sh
 tidy.sh
 tidy/*.sh
 util.sh
+util/message.sh
+util/option.sh
+util/parseopts.sh
 util/pkgbuild.sh
 util/source.sh
-- 
2.10.2

[pacman-dev] [PATCH 1/1] Introduce a 'disable-download-timeout' option

2016-12-02 Thread Christian Hesse 
From: Christian Hesse 

Add command line option ('--disable-download-timeout') and config file
option ('DisableDownloadTimeout') to disable defaults for low speed
limit and timeout on downloads. Use this if you have issues downloading
files with proxy and/or security gateway.

Signed-off-by: Christian Hesse 
---
 doc/pacman.8.txt  |  4 
 doc/pacman.conf.5.txt |  4 
 lib/libalpm/alpm.h|  2 ++
 lib/libalpm/dload.c   |  6 --
 lib/libalpm/handle.c  | 10 ++
 lib/libalpm/handle.h  |  1 +
 src/pacman/conf.c |  4 
 src/pacman/conf.h |  4 +++-
 src/pacman/pacman.c   |  6 ++
 9 files changed, 38 insertions(+), 3 deletions(-)

diff --git a/doc/pacman.8.txt b/doc/pacman.8.txt
index 0fa727e..b761560 100644
--- a/doc/pacman.8.txt
+++ b/doc/pacman.8.txt
@@ -193,6 +193,10 @@ Options
 *\--confirm*::
Cancels the effects of a previous '\--noconfirm'.
 
+*\--disable-download-timeout*::
+   Disable defaults for low speed limit and timeout on downloads. Use this
+   if you have issues downloading files with proxy and/or security gateway.
+
 
 Transaction Options (apply to '-S', '-R' and '-U')
 --
diff --git a/doc/pacman.conf.5.txt b/doc/pacman.conf.5.txt
index c665870..f92ee20 100644
--- a/doc/pacman.conf.5.txt
+++ b/doc/pacman.conf.5.txt
@@ -209,6 +209,10 @@ Options
Displays name, version and size of target packages formatted
as a table for upgrade, sync and remove operations.
 
+*DisableDownloadTimeout*::
+   Disable defaults for low speed limit and timeout on downloads. Use this
+   if you have issues downloading files with proxy and/or security gateway.
+
 
 Repository Sections
 ---
diff --git a/lib/libalpm/alpm.h b/lib/libalpm/alpm.h
index 2d2491d..1166385 100644
--- a/lib/libalpm/alpm.h
+++ b/lib/libalpm/alpm.h
@@ -926,6 +926,8 @@ int alpm_option_set_local_file_siglevel(alpm_handle_t 
*handle, int level);
 int alpm_option_get_remote_file_siglevel(alpm_handle_t *handle);
 int alpm_option_set_remote_file_siglevel(alpm_handle_t *handle, int level);
 
+int alpm_option_set_disable_dl_timeout(alpm_handle_t *handle, unsigned short 
disable_dl_timeout);
+
 /** @} */
 
 /** @addtogroup alpm_api_databases Database Functions
diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c
index 9d80358..28f1990 100644
--- a/lib/libalpm/dload.c
+++ b/lib/libalpm/dload.c
@@ -305,8 +305,10 @@ static void curl_set_handle_opts(struct dload_payload 
*payload,
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(curl, CURLOPT_XFERINFOFUNCTION, dload_progress_cb);
curl_easy_setopt(curl, CURLOPT_XFERINFODATA, (void *)payload);
-   curl_easy_setopt(curl, CURLOPT_LOW_SPEED_LIMIT, 1L);
-   curl_easy_setopt(curl, CURLOPT_LOW_SPEED_TIME, 10L);
+   if(!handle->disable_dl_timeout) {
+   curl_easy_setopt(curl, CURLOPT_LOW_SPEED_LIMIT, 1L);
+   curl_easy_setopt(curl, CURLOPT_LOW_SPEED_TIME, 10L);
+   }
curl_easy_setopt(curl, CURLOPT_HEADERFUNCTION, dload_parseheader_cb);
curl_easy_setopt(curl, CURLOPT_WRITEHEADER, (void *)payload);
curl_easy_setopt(curl, CURLOPT_NETRC, CURL_NETRC_OPTIONAL);
diff --git a/lib/libalpm/handle.c b/lib/libalpm/handle.c
index 28e8148..f760748 100644
--- a/lib/libalpm/handle.c
+++ b/lib/libalpm/handle.c
@@ -844,4 +844,14 @@ int SYMEXPORT 
alpm_option_get_remote_file_siglevel(alpm_handle_t *handle)
}
 }
 
+int SYMEXPORT alpm_option_set_disable_dl_timeout(alpm_handle_t *handle,
+   unsigned short disable_dl_timeout)
+{
+   CHECK_HANDLE(handle, return -1);
+#ifdef HAVE_LIBCURL
+   handle->disable_dl_timeout = disable_dl_timeout;
+#endif
+   return 0;
+}
+
 /* vim: set noet: */
diff --git a/lib/libalpm/handle.h b/lib/libalpm/handle.h
index 652f17d..469aecc 100644
--- a/lib/libalpm/handle.h
+++ b/lib/libalpm/handle.h
@@ -60,6 +60,7 @@ struct __alpm_handle_t {
 #ifdef HAVE_LIBCURL
/* libcurl handle */
CURL *curl; /* reusable curl_easy handle */
+   unsigned short disable_dl_timeout;
 #endif
 
 #ifdef HAVE_LIBGPGME
diff --git a/src/pacman/conf.c b/src/pacman/conf.c
index d8d64fb..de7715c 100644
--- a/src/pacman/conf.c
+++ b/src/pacman/conf.c
@@ -498,6 +498,8 @@ static int _parse_options(const char *key, char *value,
config->color = isatty(fileno(stdout)) ? 
PM_COLOR_ON : PM_COLOR_OFF;
enable_colors(config->color);
}
+   } else if(strcmp(key, "DisableDownloadTimeout") == 0) {
+   config->disable_dl_timeout = 1;
} else {
pm_printf(ALPM_LOG_WARNING,
_("config file %s, line %d: directive 
'%s' in section '%s' not recognized.\n"),
@@ -815,6 +817,8 @@ static int setup_libalpm(void)