RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
> I still don't like the idea of having a "global" system level. We can easily
> think of a system where you run stuff multi-threaded, and then you might want
> one "system" for each thread's app level.
But this I guess what application can relatively easy provide.
For instance, some global "system" callback implementation may access
application-specific thread local contexts or use other thread synchronization
mechanisms specific to that application.
I guess libcurl should only provide a thread-safe mechanism to set "system"
callbacks for apps that want to use them, and problems with invoking those
callbacks in multiple threads should be application concern.
> I also like the thought that this would have to be explicit and not implicit:
> meaning that the app would have to agree to and opt-in being overrided by a
> (specific) system level. Possibly by a CURLOPT_SYSTEM=[system handle] option,
> possibly by having a new curl_easy_init() version that takes the [system
> handle] as an argument.
Sure, it has to be explicit "opt-in" mechanism.
But the most important feature of such mechanism is that it should allow
applications providing "hosting system" for other components using libcurl
(which may be closed for modification) to be able to tune
(or even limit) their behavior when they are running in the "hosting"
environment.
So, this mechanism should specify the behavior of the libcurl "system", which
should be allowed to be initialized only once - by the "hosting" application,
and any secondary "system" tuning calls from the "hosted" components will be
ignored to avoid system "hacking".
I thought that "struct curl_system_funcs" with curl_global_init_system()
could be used for creating a "system", but may be instead we can add a new
handle type
- "system handle", as you called it, and have a set of "system options".
Something like "CURLS" or "CURLSYS" handle type for "system option" (in
addition to "easy", "multi", "share" and "url" types ) and provide
"curl_system_init()", " curl_share_setopt()" etc.
The "system" handle will represent a structure similar to curl_system_funcs
exposing its members via "system" options rather than via direct member access.
I guess "CURLOPT_SYSTEM=[system handle]" in your comment matches that idea.
But in order this to work for "hosting" systems, we will probably still need
some global "pre-transfer" callback, which will allow to set CURLOPT_SYSTEM
option early on,
before upper client code can set any options and thus override it.
> If we would do this with callbacks, then maybe there should be a system
> callback for any option the app sets.
> To allow the system to filter or alter what is actually set in the app handle.
That would be great! We can probably have such "on_option_set" filter function
as one of the options of the "system" handle.
I guess a combination of a "system" handle and "pre-transfer" global callback
is a good alternative to "curl_system_funcs/ curl_global_init_system()"
approach for "system" layer implementation.
Thanks,
Dmitry Karpov
-Original Message-
From: Daniel Stenberg
Sent: Thursday, September 29, 2022 12:09 AM
To: Dmitry Karpov
Cc: libcurl development
Subject: RE: [EXTERNAL] Re: Feature request: provide ability to set a global
callback function telling libcurl if IPv6 works on the system
On Tue, 27 Sep 2022, Dmitry Karpov wrote:
> So, for tuning just a few "system" settings, I am thinking that maybe
> we can have something like:
>
> curl_global_init_system(long flags, curl_system_funcs *system_funcs);
I still don't like the idea of having a "global" system level. We can easily
think of a system where you run stuff multi-threaded, and then you might want
one "system" for each thread's app level.
I also like the thought that this would have to be explicit and not implicit:
meaning that the app would have to agree to and opt-in being overrided by a
(specific) system level. Possibly by a CURLOPT_SYSTEM=[system handle] option,
possibly by having a new curl_easy_init() version that takes the [system
handle] as an argument.
> struct curl_system_funcs {
> curl_ipv6works ipv6_works; // Function checking if IPv6 works on
> the system
If we would do this with callbacks, then maybe there should be a system
callback for any option the app sets. To allow the system to filter or alter
what is actually set in the app handle.
--
/ daniel.haxx.se
| Commercial curl support up to 24x7 is available!
| Private help, bug fixes, support, ports, new features
| https://curl.se/support.html
--
Unsubscribe: https://lists.haxx.se/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
On Tue, 27 Sep 2022, Dmitry Karpov wrote:
So, for tuning just a few "system" settings, I am thinking that maybe we can
have something like:
curl_global_init_system(long flags, curl_system_funcs *system_funcs);
I still don't like the idea of having a "global" system level. We can easily
think of a system where you run stuff multi-threaded, and then you might want
one "system" for each thread's app level.
I also like the thought that this would have to be explicit and not implicit:
meaning that the app would have to agree to and opt-in being overrided by a
(specific) system level. Possibly by a CURLOPT_SYSTEM=[system handle] option,
possibly by having a new curl_easy_init() version that takes the [system
handle] as an argument.
struct curl_system_funcs {
curl_ipv6works ipv6_works; // Function checking if IPv6 works on the system
If we would do this with callbacks, then maybe there should be a system
callback for any option the app sets. To allow the system to filter or alter
what is actually set in the app handle.
--
/ daniel.haxx.se
| Commercial curl support up to 24x7 is available!
| Private help, bug fixes, support, ports, new features
| https://curl.se/support.html
--
Unsubscribe: https://lists.haxx.se/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
I just realized that the "pre_execute_transfer(handle)" mechanism is not going
to help to handle "system" issues as we don't have a mechanism to get a value
of some specific option when transfer is set and about to run.
We have curl_easy_setopt() but not curl_easy_getopt(), so there is no easy way
to get some option value in global functions like
"pre_execute_transfer(handle)" if we just have a transfer handle there.
Asking to provide curl_easy_getopt(), in addition to pre_execute_transfer()
mechanism will be probably too much for tuning just a few "system" settings.
Also, "system" tuning may not necessarily be always done by intercepting and
changing some transfer options.
So, for tuning just a few "system" settings, I am thinking that maybe we can
have something like:
curl_global_init_system(long flags, curl_system_funcs *system_funcs);
struct curl_system_funcs {
curl_ipv6works ipv6_works; // Function checking if IPv6 works on the system
// Add more system functions here allowing to override defaults and tune
some specific system
};
This approach covers IPv6 tuning and it is general enough to add more "system"
functions in the future without a need to add more new curl_global_init_xxx()
global functions.
Thanks,
Dmitry Karpov
-Original Message-
From: curl-library On Behalf Of Dmitry
Karpov via curl-library
Sent: Monday, September 26, 2022 3:48 PM
To: Daniel Stenberg
Cc: Dmitry Karpov ; Dmitry Karpov via curl-library
Subject: RE: [EXTERNAL] Re: Feature request: provide ability to set a global
callback function telling libcurl if IPv6 works on the system
> I could imagine a concept where you create a "system" from which you make
> easy handles, as then the "system" could define things that all subsequently
> created easy handles inherit. Like IPv6 functionality.
Exactly. The "system" level is what is needed to customize the "system"
behavior for all the handles created in the libcurl application without
touching the end client application code (regardless of whether it is opened
for modification or not).
The "system" level tweaks will allow to adjust libcurl-based code to run the
best possible way for some particular system (HW platform, OS kernel etc)
trying to satisfy the "intent" expressed in the transfer setup code as much as
possible.
> I think my main problems with your current design proposal is that
> your "system level" is global (with all the problems and limitations that
> brings) and that it needs to use a callback for information that really isn't
> needed to be set by a callback.
I think that the fundamental problem I am trying to solve can be expressed as
follows:
"How to apply some settings on all handles created by the application
(including its integrated components using libcurl) without touching the code
that sets them,
because it is not always easy or feasible to change the transfer
setting code."
In my particular case, it is "IPv6 works" check which affects how the "resolve
mode" option should be applied.
But there may be some other transfer settings that also need some tuning for
some particular "system".
I guess such "system" functionality requires using global callbacks with all
problems and limitations that it brings for all those adventurous folks who
would dare to use that functionality.
And if we look at this problem from a wider perspective, then all settings can
be tuned for some system if libcurl provides a just one "system" global
callback which can examine and modify all the options set for the transfer
before libcurl executes it.
Something like "pre_execute_transfer(CURL* easy_handle, void* data)" (and maybe
"pre_execute_multi_transfer(CURLM* multi_handle, void* data) to tune
multi-transfers)" .
These callbacks or "hooks", if set, will allow applications to adjust their
"system" behavior for a wide range of options - not just for checking if "IPv6
works" without touching end client code (which they might not be able to
change).
They will also help to address some security or policy issues when some when
integrated components may use some transfer settings which are against some
policies enforced in the "hosting" applications.
What do you think about that approach?
It still introduces new global callback(s), but the flexibility that this
approach brings in allows to do a wide range of "system" customizations beyond
just IPv6.
Thanks,
Dmitry Karpov
-Original Message-
From: Daniel Stenberg
Sent: Monday, September 26, 2022 2:52 PM
To: Dmitry Karpov
Cc: Dmitry Karpov via curl-library
Subject: RE: [EXTERNAL] Re: Feature request: provide ability to set a global
RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
> I could imagine a concept where you create a "system" from which you make > easy handles, as then the "system" could define things that all subsequently > created easy handles inherit. Like IPv6 functionality. Exactly. The "system" level is what is needed to customize the "system" behavior for all the handles created in the libcurl application without touching the end client application code (regardless of whether it is opened for modification or not). The "system" level tweaks will allow to adjust libcurl-based code to run the best possible way for some particular system (HW platform, OS kernel etc) trying to satisfy the "intent" expressed in the transfer setup code as much as possible. > I think my main problems with your current design proposal is that your > "system level" is global (with all the problems and limitations that brings) > and that it needs to use a callback for information that really isn't needed > to be set by a callback. I think that the fundamental problem I am trying to solve can be expressed as follows: "How to apply some settings on all handles created by the application (including its integrated components using libcurl) without touching the code that sets them, because it is not always easy or feasible to change the transfer setting code." In my particular case, it is "IPv6 works" check which affects how the "resolve mode" option should be applied. But there may be some other transfer settings that also need some tuning for some particular "system". I guess such "system" functionality requires using global callbacks with all problems and limitations that it brings for all those adventurous folks who would dare to use that functionality. And if we look at this problem from a wider perspective, then all settings can be tuned for some system if libcurl provides a just one "system" global callback which can examine and modify all the options set for the transfer before libcurl executes it. Something like "pre_execute_transfer(CURL* easy_handle, void* data)" (and maybe "pre_execute_multi_transfer(CURLM* multi_handle, void* data) to tune multi-transfers)" . These callbacks or "hooks", if set, will allow applications to adjust their "system" behavior for a wide range of options - not just for checking if "IPv6 works" without touching end client code (which they might not be able to change). They will also help to address some security or policy issues when some when integrated components may use some transfer settings which are against some policies enforced in the "hosting" applications. What do you think about that approach? It still introduces new global callback(s), but the flexibility that this approach brings in allows to do a wide range of "system" customizations beyond just IPv6. Thanks, Dmitry Karpov -Original Message----- From: Daniel Stenberg Sent: Monday, September 26, 2022 2:52 PM To: Dmitry Karpov Cc: Dmitry Karpov via curl-library Subject: RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system On Mon, 26 Sep 2022, Dmitry Karpov wrote: > Your proposal for fixing problem with the default "IPv6 works" > function is to move the problem from the "system" level to the > "client" level and make clients aware of which "platform" (HW, kernel > version etc) they may running on and use platform-specific "IPv6 > works" function in all places where transfer is set. I think your idea of two separate levels, system and client, is interesting, and it could be worth exploring. I could imagine a concept where you create a "system" from which you make easy handles, as then the "system" could define things that all subsequently created easy handles inherit. Like IPv6 functionality. system = curl_system_init(); easy = curl_easy_init_system(system); which possibly could be faked pretty good today with: system = curl_easy_init() easy = curl_easy_duphandle(system); an alernative could be to (somehow) store easy options for a specified scope within a multi handle, so when you create new easy handles for use in a certain multi handle you could: easy = curl_easy_inherit(multi); I think my main problems with your current design proposal is that your "system level" is global (with all the problems and limitations that brings) and that it needs to use a callback for information that really isn't needed to be set by a callback. Details that are the direct results of your introduciton of these separated levels without libcurl actually having any such separation of priviledges and now you have to sh
RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
On Mon, 26 Sep 2022, Dmitry Karpov wrote: Your proposal for fixing problem with the default "IPv6 works" function is to move the problem from the "system" level to the "client" level and make clients aware of which "platform" (HW, kernel version etc) they may running on and use platform-specific "IPv6 works" function in all places where transfer is set. I think your idea of two separate levels, system and client, is interesting, and it could be worth exploring. I could imagine a concept where you create a "system" from which you make easy handles, as then the "system" could define things that all subsequently created easy handles inherit. Like IPv6 functionality. system = curl_system_init(); easy = curl_easy_init_system(system); which possibly could be faked pretty good today with: system = curl_easy_init() easy = curl_easy_duphandle(system); an alernative could be to (somehow) store easy options for a specified scope within a multi handle, so when you create new easy handles for use in a certain multi handle you could: easy = curl_easy_inherit(multi); I think my main problems with your current design proposal is that your "system level" is global (with all the problems and limitations that brings) and that it needs to use a callback for information that really isn't needed to be set by a callback. Details that are the direct results of your introduciton of these separated levels without libcurl actually having any such separation of priviledges and now you have to shoehorn them in and then it can't be done in a very clean way. And if documenting, publishing etc is a hassle then can we make it "undocumented API" then? No, that's just worse. We either have an API or we do not. If we have it, we support it completely and it needs to be fully documented so that everyone knows what to expect from it, and it needs to be tested. Otherwise it will lead to security problems and that we accidentally slightly break behavior in a future with the following sad faces. Every function we add has a weight and a cost. It adds code, it adds docs, it adds tests and it adds to the haystack for people trying to find their needles. It is our duty to always resist adding more. Until we are convinced we need to add the new stuff. -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
> Closed source applications that use libcurl cannot be fixed unless the > creators fix their applications. This goes not only for IPv4/IPv6 issues but > for all options libcurl provides. I am not talking about applications like executables. I am referring "components", which may be integrated and run in some "hosting" application. Such architectures are quite often for embedded designs. The "hosting" application serves as a system management host (to some extent it may provide OS kind of layer on top of the underlying low-level OS-es like Embedded Linux) which sets memory and network parameters specific for the hardware platform on which it operates and provides secure and optimized environment for other "components" to run. These components are not "tightly coupled" with the host management application, so they are not aware of any memory or network management which the "host" application is doing for them. Your proposal for fixing problem with the default "IPv6 works" function is to move the problem from the "system" level to the "client" level and make clients aware of which "platform" (HW, kernel version etc) they may running on and use platform-specific "IPv6 works" function in all places where transfer is set. But there are fundamental problems with this approach, which makes it not a good solution. 1. What if there are more than 10+ platforms which require platform-specific handling of the "IPv6 works" function? Then the client code should do some pseudo-code: if (platform == platform1) ... use platform1 "IPv6 works" ... if (platform == platform10) ... use platform10 "IPv6 works" And what if in future the same code needs to run on 10 more platforms? The client code needs to be updated with any new platform then. 2. How to "inject dependency" on platform specific low-level "IPv6 works" function into the client code? For example, some "components" may be using multiple layers and high-level languages (i.e. JavaScript etc), and the transfer setup code (which is done on some low-level layer) is not exposed for the upper layers at all. Such components provide system-agnostic application frameworks, which share the same code for different OSes (i.e. Windows, MacOS, Linux), and they are also agnostic to which libcurl (single-stack or dual-stack) their lower layer may be using. So, for such components there will be no way to inject "platform A" specific "IPv6 works" function and modify their transfer setup code on the upper "client" level. > Because it introduces both a new public API function call plus a new concept > of a global callback. This new public API is proposed only for providing a good and convenient solution for the "system" issues mentioned above. It is by no means a new way to set the "resolve mode" option for transfers. On the contrary, it emphasizes that "IPv6 works" is a system setting, not a transfer setting, and it aims to make transfer setup code agnostic to the fact that IPv6 doesn't work on some specific platform. In other words, why some libcurl-based code should be aware that it is using dual-stack libcurl and on some platform "IPv6 works" check creates unnecessary delays, and transfer setup code in client code should be modified to handle these platform-specific delays? If "IPv6 works" is a system setting which depends on platforms where libcurl is running, then it is quite logical to have it as a global callback. Such approach will allow to avoid any problems with handling platform-specific IPv6 issues in the client code and make client code "loosely coupled" with the system management process running on that platform. I see the same approach was used for memory management functions, which are also "system" level functionality, so I am not inventing a completely new approach for system-level customization . I understand your reluctance to add a new public API, document it etc. But considering benefits it provides for dealing with dual-stack issues on embedded platforms (where a very wide range of linux kernels, including custom builds from SOC vendors), I think it is worth it. And if documenting, publishing etc is a hassle then can we make it "undocumented API" then? It is still will be a useful feature to have for folks who are facing issues with the default "IPv6 works" in embedded world? Thanks, Dmitry Karpov -Original Message- From: Daniel Stenberg Sent: Saturday, September 24, 2022 2:16 PM To: Dmitry Karpov Cc: Dmitry Karpov via curl-library Subject: RE: [EXTERNAL] Re: Feature request: provi
RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
On Fri, 23 Sep 2022, Dmitry Karpov wrote: But how do you suggest to address the issues with "closed code" as I outlined in my previous post? " - if you code integrates (or will do that future) some "3rd party" components which might be using libcurl and dual-stack resolve modes, which you can't modify" Closed source applications that use libcurl cannot be fixed unless the creators fix their applications. This goes not only for IPv4/IPv6 issues but for all options libcurl provides. I don't understand why do you call my proposal "a new way"? Because it introduces both a new public API function call plus a new concept of a global callback. -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
Daniel, I think I am repeating my arguments because some problems that you solution brings in are not addressed. Even it is very painful to change transfer setup code in all the places to utilize your fix for IPv4 mode, it is at least doable. But how do you suggest to address the issues with "closed code" as I outlined in my previous post? " - if you code integrates (or will do that future) some "3rd party" components which might be using libcurl and dual-stack resolve modes, which you can't modify" The integrated code if it uses dual-stack resolve modes cannot be covered by your solution and connection delays will remain there. Such code cannot be always modified, especially if it comes in binaries. I have such code in my applications, and someone else may have such code as well. Also, how do you suggest to address the problem I outlined in my previous post which your PR introduced? "This "lazy" approach doesn't allow to create a multi-handle ahead of time to avoid delays on dual-stack connections." The previous mechanism at least allowed to do that to mitigate the Curl_ipv6works() delays, but not any more after the PR is integrated. > I have not been convinced that we need to add a new way to say IPv6 doesn't > work. I don't understand why do you call my proposal "a new way"? It is fundamentally the same old way with an extended option to provide an application-specific way to detect that IPv6 doesn't work on the system. I guess your PR can be called a "new way" as the penalty for Curl_ipv6works() delays from now on will always be paid for dual-stack connections unless someone will implement this cumbersome mechanism of applying custom "IPv6 works" detection on every dual-stack transfer setup if it is possible (and still have delays when it is not possible to modify the code). I don't want to argue about simplicity of both approaches, but I think that side-by-side comparison really tells which approach is simpler and easier to use. Thanks, Dmitry Karpov -Original Message- From: Daniel Stenberg Sent: Friday, September 23, 2022 3:10 PM To: Dmitry Karpov Cc: Dmitry Karpov via curl-library Subject: RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system On Fri, 23 Sep 2022, Dmitry Karpov wrote: > This would create the same outcome ONLY after your PR fixing IPv4 mode is > applied. Source code changes tend to work like that: they only have an effect after they are actually applied. And my PR is already merged. I'm sorry, but I think you are repeating your arguments and while I could repeat my arguments again as a reponse I don't think it would drive the discussion forward. I have not been convinced that we need to add a new way to say IPv6 doesn't work. -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
On Fri, 23 Sep 2022, Dmitry Karpov wrote: This would create the same outcome ONLY after your PR fixing IPv4 mode is applied. Source code changes tend to work like that: they only have an effect after they are actually applied. And my PR is already merged. I'm sorry, but I think you are repeating your arguments and while I could repeat my arguments again as a reponse I don't think it would drive the discussion forward. I have not been convinced that we need to add a new way to say IPv6 doesn't work. -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
In addition to my previous arguments, a few comments about Daniel's PR (https://github.com/curl/curl/pull/9553). While this PR helps IPv4-only mode to avoid delays caused by default Curl_ipv6works(), it eliminates some workarounds for dual-stack modes, which were available before the PR. The current approach calls Curl_ipv6works() when a multi-handle is created and before any easy handle is added to it. This allows applications to create a multi handle ahead of time and consume the Curl_ipv6works() delays not in time critical areas - potentially long before easy handles using dual-stack are added. Then easy handles will not have connection delays in time critical areas due to Curl_ipv6works(), because it was already called during multi-handle creation. Such approach maybe not easy to implement in all the cases, but at least it was available. The PR moves the Curl_ipv6works() call to the time when the first dual-stack connection is established for the multi-handle. This "lazy" approach doesn't allow to create a multi-handle ahead of time to avoid delays on dual-stack connections. So, if someone was using this approach it will not work anymore after the PR is integrated. I fully support the PR, but I also think that this is also another argument for adding the "IPv6 works" system callback. Let's say someone doesn't want to change the Curl_ipv6works() default behavior, but he/she wants to have "IPv6 works on the system" check ahead of time. So, when it is time to make dual-stack connections, the check (even if takes 30+ms) is already done, and the connection doesn't need to call Curl_ipv6works() in time-critical area. The "IPv6 works" system callback will allow easily to achieve that. Someone can do the default check once at some time and then set a global flag used in the callback thus eliminating any delays after the initial check. Or maybe implement a more flexible approach when these checks are done periodically - by timer or when application detects some system configuration changes (i.e. network changes) and update the "IPv6 works" flag. In other words, the "IPv6 works" system callback provides a nice flexibility for implementing different scenarios. Thanks, Dmitry Karpov From: curl-library On Behalf Of Dmitry Karpov via curl-library Sent: Friday, September 23, 2022 1:35 PM To: libcurl development Cc: Dmitry Karpov Subject: RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system I definitely agree with the observations, but not with the alternative approach. Using a file like “. /etc/default/libcurl” will create more issues rather than helping. It is like using a global “IPv6 works” variable, which was set during curl_global_init() in the past. That approach was prone to multi-threading issues and race conditions when network settings affecting IPv6 stack were set after curl_global_init(), and adding reading/wring to this “config” file (which has to be done somewhere) from different places just adds more problems. It will be difficult to avoid file access issues if the code detecting if IPv6 works, needs to update this file when it is being read by libcurl. So, I think that the “config” file approach is much more complex and has more implementation difficulties (i.e. just dealing with file names on Windows) than the approach using “IPv6 works” system callback. Thanks, Dmitry Karpov From: curl-library mailto:curl-library-boun...@lists.haxx.se>> On Behalf Of Timothe Litt via curl-library Sent: Friday, September 23, 2022 1:15 PM To: curl-library@lists.haxx.se<mailto:curl-library@lists.haxx.se> Cc: Timothe Litt mailto:l...@acm.org>> Subject: Re: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system This discussion revolves around some platforms on which it is difficult to efficiently determine if IPv6 is *not* available. Observations: * The proposed solutions involve determining the converse. * On a given platform, answer is likely to be stable for a very long time. E.g. until a kernel/network/system change * The OP reports that getting the "no IPv6" result efficiently is very situation - and probably customer - dependent - e.g. is using heuristics that would not be a good fit in libcurl * libcurl isn't the right place for customer-dependent work-arounds; at best the detection would be once/process Here's an alternative approach to consider: * Have libcurl read an optional configuration file; e.g. /etc/default/libcurl * The initial contents would be 'ipv6_disable:1" * When a similar issue arises for IPv4, or DNS, codesets, or some other system behavior, it's easy to add controls * If the file doesn't exist, libcurl uses its c
RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
I definitely agree with the observations, but not with the alternative approach. Using a file like “. /etc/default/libcurl” will create more issues rather than helping. It is like using a global “IPv6 works” variable, which was set during curl_global_init() in the past. That approach was prone to multi-threading issues and race conditions when network settings affecting IPv6 stack were set after curl_global_init(), and adding reading/wring to this “config” file (which has to be done somewhere) from different places just adds more problems. It will be difficult to avoid file access issues if the code detecting if IPv6 works, needs to update this file when it is being read by libcurl. So, I think that the “config” file approach is much more complex and has more implementation difficulties (i.e. just dealing with file names on Windows) than the approach using “IPv6 works” system callback. Thanks, Dmitry Karpov From: curl-library On Behalf Of Timothe Litt via curl-library Sent: Friday, September 23, 2022 1:15 PM To: curl-library@lists.haxx.se Cc: Timothe Litt Subject: Re: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system This discussion revolves around some platforms on which it is difficult to efficiently determine if IPv6 is *not* available. Observations: * The proposed solutions involve determining the converse. * On a given platform, answer is likely to be stable for a very long time. E.g. until a kernel/network/system change * The OP reports that getting the "no IPv6" result efficiently is very situation - and probably customer - dependent - e.g. is using heuristics that would not be a good fit in libcurl * libcurl isn't the right place for customer-dependent work-arounds; at best the detection would be once/process Here's an alternative approach to consider: * Have libcurl read an optional configuration file; e.g. /etc/default/libcurl * The initial contents would be 'ipv6_disable:1" * When a similar issue arises for IPv4, or DNS, codesets, or some other system behavior, it's easy to add controls * If the file doesn't exist, libcurl uses its current defaults/behaviors - configure and runtime * Setting up the file becomes an installation-time issue on your platform. It can take as long as you like, and be as system specific as you like. * The overhead for libcurl is very small. On most systems, the file won't exist - and as strace will tell you, many, many files are looked-up and ignored when any non-trivial process starts. If the file does exist, the flags can be cached when the file is parsed. If performance ever becomes an issue (seems unlikely), the cache could be in a shared memory segment. * Optionally, libcurl could provide an API function to query a flag by name. This would seem to meet the OP's requirements: no change to the application, easy to install on the affected platforms, efficient. And it is minimally disruptive to libcurl, which should satisfy the developers. Timothe Litt ACM Distinguished Engineer -- This communication may not represent the ACM or my employer's views, if any, on the matters discussed. On 23-Sep-22 15:15, Dmitry Karpov via curl-library wrote: Let me reply a bit out of order and combine some arguments: Exactly. Because it would create exactly the same outcome. This would create the same outcome ONLY after your PR fixing IPv4 mode is applied. Before that PR, the problem existed for ALL modes, so there was no work around even for IPv4 mode. First: most developers won't write this kind of function at all. curl has supported IPv6 since January 2001 and we are talking about doing this in a future release - after nearly 22 years of shipping IPv6 suppport. Clearly a lot of users manage decently without this. Yes, it may be supported since a while ago, but when I started migrating to dual-stack more than a year ago, I stepped on multiple issues, which made me think that dual-stack has been hardly used on a wide scale where performance was quite critical - at least on embedded platforms with large populations. My applications run on different HW platforms, use a wide range of linux kernels with different configurations and user population is greater than 62M, which covers a very wide range of ISPs, routers and network setups. That's why maybe I stepped on the issues which other didn't see - or just didn't care. The 30ms delay during connection setup maybe not a critical issue for everyone, but it is a still an issue and regression when migrating from a single-stack for time critical applications. .They use the same code to determine IPv6 functionality: your function, and they do it roughly at the same time in the process: right before setting up a new connection. The only d
Re: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
ti handle is created and then its result is applied to every easy handle added to that multi-handle - regardless the resolve mode set on the easy handle. (And a while ago that check was done only once in curl_global_init() which created a base for multi-threading and race condition issues) What you suggest is essentially to consider "IPv6 doesn't work on a system" as a transfer level detail and do such check for every transfer setup. And this just goes against consistency and simplicity, in my opinion. So, if someone steps on the same connection delay issues using dual-stack libcurl as I did - your recommendation (again after you PR is integrated) would be: Because our default "IPv6 works" check doesn't work for you do the following: - write your own function doing " IPv6 works" check which works better for you (the same step as in my approach) - change ALL the places in the code where your transfers are set up, invoke your "IPv6 works" check function and force your transfers to use IPv4 resolve mode if that check fails. Fine print details: - be careful not to miss any single place where transfers might be using dual-stack in your code. - when you add new transfers to your code be careful not to miss the "IPv6 works" check function if they might be using dual-stack. - if you code integrates (or will do that future) some "3rd party" components which might be using libcurl and dual-stack resolve modes, which you can't modify then, sorry, you are out of luck. This is an "engineering issue", so the connection delays when "IPv6 doesn't work" will be there. With my approach: Because our default "IPv6 works" check doesn't work for you do the following: - write your own function doing "IPv6 works" check which works better for you. - set your function as "IPv6 works" callback in curl_global_init_ipv6() Fine print details: None. It will work for current and future code using dual-stack modes and for "closed code" of 3rd party components without any modifications in the transfer setup code. So, I guess from this side-by-side comparison it is pretty obvious which approach adheres more to the principles of consistency and simplicity. Also, there are already "system" level callbacks which allow to set system wide memory management functions in curl_global_init_mem(). And I just propose to add a similar system wide "IPv6 works on the system" check callback, so I am quite consistent here with what libcurl does. Thanks, Dmitry Karpov -Original Message- From: Daniel Stenberg Sent: Friday, September 23, 2022 12:54 AM To: Dmitry Karpov Cc: Dmitry Karpov via curl-library Subject: RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system On Thu, 22 Sep 2022, Dmitry Karpov wrote: If I understand what you suggest, then in all the places where the "AUTO" (CURL_IPRESOLVE_WHATEVER) is used, I would need to use something like: long resolve_mode = my_ipv6_works() ? CURL_IPRESOLVE_WHATEVER : CURL_IPRESOLVE_V4; curl_easy_setopt(curl, CURLOPT_IPRESOLVE, resolve_mode); Exactly. Because it would create exactly the same outcome. First one is that the "AUTO" will stop being the most efficient mode for dual-stack, because its efficiency will depend on how "IPv6 works" check (provided by "factory default" Curl_ipv6works()) works on some systems (kernels, network configs etc). Not at all. You say you can write a "ipv6works" function (as a callback) that can detect/set the status, so clearly then it can be called by your own application as well. There is no difference in effectiveness between these two approaches. They use the same code to determine IPv6 functionality: your function, and they do it roughly at the same time in the process: right before setting up a new connection. The only difference is how that function is called. And if some curl application accidentally steps on this issue and needs to implement some custom "IPv6 works" mechanism, then it is suggested that after implementing it, application developer should go through all the places where transfer options are set, explicitly use application specific "IPv6 works" function and explicitly select IPv4 mode if that function returns "false". Exactly like you do when you set other properties and control other characteristcs of libcurl transfers. There are *300* options for curl_easy_setopt(), I'm sure you realized by now that this is the libcurl API model. For some reason you think this particular transfer detail deserves a completely new and different take, and this reason seems to be because
RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
ails: None. It will work for current and future code using dual-stack modes and for "closed code" of 3rd party components without any modifications in the transfer setup code. So, I guess from this side-by-side comparison it is pretty obvious which approach adheres more to the principles of consistency and simplicity. Also, there are already "system" level callbacks which allow to set system wide memory management functions in curl_global_init_mem(). And I just propose to add a similar system wide "IPv6 works on the system" check callback, so I am quite consistent here with what libcurl does. Thanks, Dmitry Karpov -Original Message- From: Daniel Stenberg Sent: Friday, September 23, 2022 12:54 AM To: Dmitry Karpov Cc: Dmitry Karpov via curl-library Subject: RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system On Thu, 22 Sep 2022, Dmitry Karpov wrote: > If I understand what you suggest, then in all the places where the "AUTO" > (CURL_IPRESOLVE_WHATEVER) is used, I would need to use something like: > > long resolve_mode = my_ipv6_works() ? CURL_IPRESOLVE_WHATEVER : > CURL_IPRESOLVE_V4; curl_easy_setopt(curl, CURLOPT_IPRESOLVE, > resolve_mode); Exactly. Because it would create exactly the same outcome. > First one is that the "AUTO" will stop being the most efficient mode > for dual-stack, because its efficiency will depend on how "IPv6 works" > check (provided by "factory default" Curl_ipv6works()) works on some > systems (kernels, network configs etc). Not at all. You say you can write a "ipv6works" function (as a callback) that can detect/set the status, so clearly then it can be called by your own application as well. There is no difference in effectiveness between these two approaches. They use the same code to determine IPv6 functionality: your function, and they do it roughly at the same time in the process: right before setting up a new connection. The only difference is how that function is called. > And if some curl application accidentally steps on this issue and > needs to implement some custom "IPv6 works" mechanism, then it is > suggested that after implementing it, application developer should go > through all the places where transfer options are set, explicitly use > application specific > "IPv6 works" function and explicitly select IPv4 mode if that function > returns "false". Exactly like you do when you set other properties and control other characteristcs of libcurl transfers. There are *300* options for curl_easy_setopt(), I'm sure you realized by now that this is the libcurl API model. For some reason you think this particular transfer detail deserves a completely new and different take, and this reason seems to be because of the way you have designed and setup your libcurl usage. In my view, it does not seem to be particularly well founded from a libcurl architectural point of view. I'm sorry, but I think consistency and simplicity is more important here. > Frankly, I think it is just cruel to make developers do that when > everything can be solved in just one system level "IPv6 works" callback. First: most developers won't write this kind of function at all. curl has supported IPv6 since January 2001 and we are talking about doing this in a future release - after nearly 22 years of shipping IPv6 suppport. Clearly a lot of users manage decently without this. Then, many users do their setopt logic in a single place or at least offers mechanics to extend the set without going through hoops. Doing this logic as a callback or as a setopt then means roughly the same effort. I don't think insisting on consistency is cruel. > And for me the biggest problem is that I just can't change the code of > certain curl-based components used in my application. They are written > by some other developers and closed for any modifications. This is the core of your issue, not necessarily related to the libcurl issue at hand. I would argue that this is a fragile way to go about libcurl use (and fix). What if you find another flaw tomorrow where a setopt is used wrongly? Would we solve those as well with additional work-arounds and coming in through the backdoor just because someone else has the key to the front door? > In other words, it is much easier and cleaner to override how > Curl_ipv6works() to adjust to demands of certain systems/scenarios > than do checks if "IPv6 works" and change resolve mode in every place > where application sets a transfer. Easier and cleaner for you. I maintain that it is dirtier for libcurl and its API (design). > Also, Curl_ipv6works() is called only once per multi-ha
RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
Unfortunately, my code detecting whether IPv6 works on my systems is not a generic one and very specific to linux kernels, network configurations and software used in my devices. It also depends on some other components, which just cannot be integrated or used inside libcurl. That's why I suggested to have a system callback which would allow developers to provide their own "IPv6 works" detection mechanism overriding the default one. Thanks, Dmitry Karpov -Original Message- From: curl-library On Behalf Of Daniel F via curl-library Sent: Friday, September 23, 2022 9:41 AM To: libcurl development Cc: dan...@poradnik-webmastera.com Subject: Re: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system > In other words, it is much easier and cleaner to override how > Curl_ipv6works() to adjust to demands of certain systems/scenarios > than do checks if "IPv6 works" and change resolve mode in every place > where application sets a transfer. Maybe you could contribute your system-specific IPv6 detection code as patch(es) for libcurl? This approach would not require any new callback. Regards, Daniel -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
In other words, it is much easier and cleaner to override how Curl_ipv6works() to adjust to demands of certain systems/scenarios than do checks if "IPv6 works" and change resolve mode in every place where application sets a transfer. Maybe you could contribute your system-specific IPv6 detection code as patch(es) for libcurl? This approach would not require any new callback. Regards, Daniel -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
On Thu, Sep 22, 2022 at 11:48:58PM +, Dmitry Karpov via curl-library wrote: > And for me the biggest problem is that I just can't change the code of > certain curl-based components used in my application. > They are written by some other developers and closed for any modifications. That is what I suspected, and it's why I described the underlying issue as a "software engineering" one. Dan -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
On Thu, 22 Sep 2022, Dmitry Karpov wrote: If I understand what you suggest, then in all the places where the "AUTO" (CURL_IPRESOLVE_WHATEVER) is used, I would need to use something like: long resolve_mode = my_ipv6_works() ? CURL_IPRESOLVE_WHATEVER : CURL_IPRESOLVE_V4; curl_easy_setopt(curl, CURLOPT_IPRESOLVE, resolve_mode); Exactly. Because it would create exactly the same outcome. First one is that the "AUTO" will stop being the most efficient mode for dual-stack, because its efficiency will depend on how "IPv6 works" check (provided by "factory default" Curl_ipv6works()) works on some systems (kernels, network configs etc). Not at all. You say you can write a "ipv6works" function (as a callback) that can detect/set the status, so clearly then it can be called by your own application as well. There is no difference in effectiveness between these two approaches. They use the same code to determine IPv6 functionality: your function, and they do it roughly at the same time in the process: right before setting up a new connection. The only difference is how that function is called. And if some curl application accidentally steps on this issue and needs to implement some custom "IPv6 works" mechanism, then it is suggested that after implementing it, application developer should go through all the places where transfer options are set, explicitly use application specific "IPv6 works" function and explicitly select IPv4 mode if that function returns "false". Exactly like you do when you set other properties and control other characteristcs of libcurl transfers. There are *300* options for curl_easy_setopt(), I'm sure you realized by now that this is the libcurl API model. For some reason you think this particular transfer detail deserves a completely new and different take, and this reason seems to be because of the way you have designed and setup your libcurl usage. In my view, it does not seem to be particularly well founded from a libcurl architectural point of view. I'm sorry, but I think consistency and simplicity is more important here. Frankly, I think it is just cruel to make developers do that when everything can be solved in just one system level "IPv6 works" callback. First: most developers won't write this kind of function at all. curl has supported IPv6 since January 2001 and we are talking about doing this in a future release - after nearly 22 years of shipping IPv6 suppport. Clearly a lot of users manage decently without this. Then, many users do their setopt logic in a single place or at least offers mechanics to extend the set without going through hoops. Doing this logic as a callback or as a setopt then means roughly the same effort. I don't think insisting on consistency is cruel. And for me the biggest problem is that I just can't change the code of certain curl-based components used in my application. They are written by some other developers and closed for any modifications. This is the core of your issue, not necessarily related to the libcurl issue at hand. I would argue that this is a fragile way to go about libcurl use (and fix). What if you find another flaw tomorrow where a setopt is used wrongly? Would we solve those as well with additional work-arounds and coming in through the backdoor just because someone else has the key to the front door? In other words, it is much easier and cleaner to override how Curl_ipv6works() to adjust to demands of certain systems/scenarios than do checks if "IPv6 works" and change resolve mode in every place where application sets a transfer. Easier and cleaner for you. I maintain that it is dirtier for libcurl and its API (design). Also, Curl_ipv6works() is called only once per multi-handle, and the "IPv6 works" setting is then used for all easy handles added to that multi-handle. So, essentially there is no need to do "IPv6 works" for every easy handle - that's also what the system "IPv6 works" callback efficiently allows to avoid. When you write your own ipv6works function, you can decide yourself how often and when it needs to rerun the check. There's nothing that says it needs to do anything for every easy handle, it could just return the previous result. -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
> What I cannot understand is how you can implement a working callback for this > purpose, called when the connection is to setup, but you cannot set an easy > option, done just before the connection is about to get setup. > Why would one work and not the other? Not sure if I understand what easy option are you referring to? Let's say, I have some curl transfer code that sets the "AUTO" option in many different places or it is set by default. So, with this option set, I want my transfers to use dual-stack when it is available and instantly use IPv4 when IPv6 doesn't work - and thus have the same behavior regardless of whether IPv6-enabled dual-stack libcurl or IPv4 is used. If I understand what you suggest, then in all the places where the "AUTO" (CURL_IPRESOLVE_WHATEVER) is used, I would need to use something like: long resolve_mode = my_ipv6_works() ? CURL_IPRESOLVE_WHATEVER : CURL_IPRESOLVE_V4; curl_easy_setopt(curl, CURLOPT_IPRESOLVE, resolve_mode); If so, then there are a lot of problems with this approach. First one is that the "AUTO" will stop being the most efficient mode for dual-stack, because its efficiency will depend on how "IPv6 works" check (provided by "factory default" Curl_ipv6works()) works on some systems (kernels, network configs etc). And if some curl application accidentally steps on this issue and needs to implement some custom "IPv6 works" mechanism, then it is suggested that after implementing it, application developer should go through all the places where transfer options are set, explicitly use application specific "IPv6 works" function and explicitly select IPv4 mode if that function returns "false". Frankly, I think it is just cruel to make developers do that when everything can be solved in just one system level "IPv6 works" callback. And for me the biggest problem is that I just can't change the code of certain curl-based components used in my application. They are written by some other developers and closed for any modifications. So, I will not be able to modify every place where transfers are set in my application just to force it to select the resolve mode to work around the Curl_ipv6works() issues on this specific system. And when I deploy my application on the platforms where Curl_ipv6works() works OK, it will be silly to do changes like that there. In other words, it is much easier and cleaner to override how Curl_ipv6works() to adjust to demands of certain systems/scenarios than do checks if "IPv6 works" and change resolve mode in every place where application sets a transfer. Also, Curl_ipv6works() is called only once per multi-handle, and the "IPv6 works" setting is then used for all easy handles added to that multi-handle. So, essentially there is no need to do "IPv6 works" for every easy handle - that's also what the system "IPv6 works" callback efficiently allows to avoid. Thanks, Dmitry Karpov -Original Message----- From: Daniel Stenberg Sent: Thursday, September 22, 2022 2:47 PM To: Dmitry Karpov via curl-library Cc: Dmitry Karpov Subject: RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system On Thu, 22 Sep 2022, Dmitry Karpov via curl-library wrote: > That's right. I don't think that it is possible to change the > Curl_ipv6works() implementation that it will work the same way with > zero delays for ALL possible kernels/network configurations. And it is > not possible to change all kernels where some curl application may run > to make > Curl_ipv6works() return false with 0ms delay when there are some > issues with > IPv6 on kernel levels. What I cannot understand is how you can implement a working callback for this purpose, called when the connection is to setup, but you cannot set an easy option, done just before the connection is about to get setup. Why would one work and not the other? -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
On Thu, 22 Sep 2022, Dmitry Karpov via curl-library wrote: That's right. I don't think that it is possible to change the Curl_ipv6works() implementation that it will work the same way with zero delays for ALL possible kernels/network configurations. And it is not possible to change all kernels where some curl application may run to make Curl_ipv6works() return false with 0ms delay when there are some issues with IPv6 on kernel levels. What I cannot understand is how you can implement a working callback for this purpose, called when the connection is to setup, but you cannot set an easy option, done just before the connection is about to get setup. Why would one work and not the other? -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
it is just an example of how the problems with Curl_ipv6works() can be worked around, and other developers can implement something else. Once there is a way to specify the "IP works" callback, it will provide a flexibility to handle problems with default Curl_ipv6works() using different ways -best for some specific application. > You can consider another approach for this - add new global function > curl_global_setopt. It would better fit how libcurl works now. This new > function should be called after curl_global_init/curl_global_init_mem and before any other libcurl calls. No, global options are not going to help here. Each transfer can select any mode: IPv4, IPv6 or AUTO - no matter whether it comes from local and global options. And for IPv6 or AUTO modes, the Curl_ipv6works() will always be called with potential connection or connection failure delays. And as I mentioned above, because it is not feasible to fix IPv6-related issues causing the delays in all possible kernels where some curl application may be running, there is a need for workaround which the system "IPv6 works" callback can provide. Thanks, Dmitry Karpov -Original Message- From: curl-library On Behalf Of Daniel Stenberg via curl-library Sent: Thursday, September 22, 2022 8:33 AM To: Daniel F via curl-library Cc: Daniel Stenberg Subject: Re: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system On Thu, 22 Sep 2022, Daniel F via curl-library wrote: > Yes, this can cause problems. One way to address this is to document > it somewhere, unfortunately people still would be able to write bad > code and complain here that something does not work. So better > approach would be to pass all these global options to curl_global_init_opts > function: I think the even better way is to resist adding global options! -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
On Thu, 22 Sep 2022, Daniel F via curl-library wrote: Yes, this can cause problems. One way to address this is to document it somewhere, unfortunately people still would be able to write bad code and complain here that something does not work. So better approach would be to pass all these global options to curl_global_init_opts function: I think the even better way is to resist adding global options! -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
W dniu 2022-09-22 15:57, Daniel Stenberg napisał(a): On Thu, 22 Sep 2022, Daniel F via curl-library wrote: You can consider another approach for this - add new global function curl_global_setopt. It would better fit how libcurl works now. This new function should be called after curl_global_init/curl_global_init_mem and before any other libcurl calls. In theory, yes. We have however rather tried to decrease the use of globals over time since they make things complicated in multi-threaded situations. Yes, this can cause problems. One way to address this is to document it somewhere, unfortunately people still would be able to write bad code and complain here that something does not work. So better approach would be to pass all these global options to curl_global_init_opts function: struct CURL_INIT_OPTIONS curl_opts; memset(&curl_opts, 0, sizeof(curl_opts)); curl_opts.flags = CURL_GLOBAL_DEFAULT; curl_opts.foo = ...; curl_opts.bar = ...; curl_global_init_opts(&curl_opts, sizeof(curl_opts)); 2nd parameter for curl_global_init_opts is added for forward compatibility, so libcurl could check if structure passed by application contains given field. Such approach is common is Windows API. -- Regards, Daniel -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
On Thu, 22 Sep 2022, Daniel F via curl-library wrote: You can consider another approach for this - add new global function curl_global_setopt. It would better fit how libcurl works now. This new function should be called after curl_global_init/curl_global_init_mem and before any other libcurl calls. In theory, yes. We have however rather tried to decrease the use of globals over time since they make things complicated in multi-threaded situations. -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
W dniu 2022-09-22 03:21, Dan Fandrich via curl-library napisał(a): I'm betting that this problem can be fixed without adding a new callback, especially one that goes against the way libcurl works, namely that callbacks are used to change behaviour during a transfer. You can consider another approach for this - add new global function curl_global_setopt. It would better fit how libcurl works now. This new function should be called after curl_global_init/curl_global_init_mem and before any other libcurl calls. In the future you could also consider replacing curl_global_init_mem with curl_global_setopt+curl_global_init calls. Regards, Daniel -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
On Thu, 22 Sep 2022, Dmitry Karpov via curl-library wrote: For instance, curl application may implement the "IPv6 works" callback based on some atomic variable, which is a magnitude faster than the socket system call done in the Curl_ipv6works(). An application that can "just use an atomic variable" can also "just set an easy option" and get the exact same effect, right? This suggestion reads to me like a work-around for you not wanting to change code in the place that would afix the problem, but you rather propose and add a new public libcurl function call so that you can do the change elsewhere in your app. Generally, having two ways to do the same thing is bad. -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
On Wed, 21 Sep 2022, Dan Fandrich via curl-library wrote: Since you're calling it a regression, then where did that regression occur? Was it in libcurl or in the kernel? Maybe this problem needs to be solved elsewhere. I guess this refers to the changing of Curl_ipv6works() to store the state in the multi handle as compared to the previous take where the state was stored globally. Done in 0b030a5b232bd9fc, shipped since 7.69.0. -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
On Wed, Sep 21, 2022 at 06:21:08PM -0700, Dan Fandrich wrote: > On Thu, Sep 22, 2022 at 12:24:57AM +, Dmitry Karpov via curl-library > wrote: > > > If Curl_ipv6works() were not called in the CURL_IPRESOLVE_V6 case, would > > > that solve the issues that are remaining? > > > > Your question was: > > "So, if libcurl eliminated that call in the CURL_IPRESOLVE_V4 case, would > > it fix your problem?" > > That was my previous question. The question above refers to CURL_IPRESOLVE_V6. > > > YES, adding a callback doesn't fix anything on its own. But it allows curl > > applications to work around problems/regressions caused by default > > Curl_ipv6works() behavior. > > I'm betting that this problem can be fixed without adding a new callback, > especially one that goes against the way libcurl works, namely that callbacks > are used to change behaviour during a transfer. You also didn't answer this question: > Since you're calling it a regression, then where did that regression occur? > Was > it in libcurl or in the kernel? Maybe this problem needs to be solved > elsewhere. -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
On Thu, Sep 22, 2022 at 12:24:57AM +, Dmitry Karpov via curl-library wrote: > > If Curl_ipv6works() were not called in the CURL_IPRESOLVE_V6 case, would > > that solve the issues that are remaining? > > Your question was: > "So, if libcurl eliminated that call in the CURL_IPRESOLVE_V4 case, would it > fix your problem?" That was my previous question. The question above refers to CURL_IPRESOLVE_V6. > YES, adding a callback doesn't fix anything on its own. But it allows curl > applications to work around problems/regressions caused by default > Curl_ipv6works() behavior. I'm betting that this problem can be fixed without adding a new callback, especially one that goes against the way libcurl works, namely that callbacks are used to change behaviour during a transfer. Dan -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
> My point is that adding a callback doesn't fix anything on its own. An > application would need to add a function to be called back, and so far, your > application is the only one that would currently use it. I suspect there > is > a better way to solve this issue. For example, you didn't answer this > question from my last message: > If Curl_ipv6works() were not called in the CURL_IPRESOLVE_V6 case, would > that solve the issues that are remaining? Your question was: "So, if libcurl eliminated that call in the CURL_IPRESOLVE_V4 case, would it fix your problem?" And I answered that question: " Daniel's PR fixed the issue only for CURL_IPRESOLVE_V4, but not for CURL_IPRESOLVE_WHATEVER and CURL_IPRESOLVE_IPV6." So, my problems will NOT be fixed if libcurl eliminated the If Curl_ipv6works() call in the CURL_IPRESOLVE_V4 case. The problems remain for CURL_IPRESOLVE_WHATEVER and CURL_IPRESOLVE_IPV6 modes. YES, adding a callback doesn't fix anything on its own. But it allows curl applications to work around problems/regressions caused by default Curl_ipv6works() behavior. Maybe, there is a better way to fix this issue, but I don't think it is possible to provide the "IP6 works" function which works for ALL possible cases - thus there is a need for customization. For instance, curl application may implement the "IPv6 works" callback based on some atomic variable, which is a magnitude faster than the socket system call done in the Curl_ipv6works(). And libcurl provides memory callbacks in a similar fashion, which can optimize memory allocations for some applications. I guess whoever proposed memory callbacks also faced arguments like "your application is the only one that would currently use it.", which can be said about any new feature. 😊 Thanks, Dmitry Karpov -Original Message- From: curl-library On Behalf Of Dan Fandrich via curl-library Sent: Wednesday, September 21, 2022 3:59 PM To: libcurl development Cc: Dan Fandrich Subject: Re: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system On Wed, Sep 21, 2022 at 10:46:45PM +, Dmitry Karpov via curl-library wrote: > I think that my proposal will provide useful customization for dual-stack > curl applications, and I am not sure that I fully understand your objections > like " It might in your situation, but it wouldn't in everybody else's.". My point is that adding a callback doesn't fix anything on its own. An application would need to add a function to be called back, and so far, your application is the only one that would currently use it. I suspect there is a better way to solve this issue. For example, you didn't answer this question from my last message: If Curl_ipv6works() were not called in the CURL_IPRESOLVE_V6 case, would that solve the issues that are remaining? Since you're calling it a regression, then where did that regression occur? Was it in libcurl or in the kernel? Maybe this problem needs to be solved elsewhere. Dan -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
On Wed, Sep 21, 2022 at 10:46:45PM +, Dmitry Karpov via curl-library wrote: > I think that my proposal will provide useful customization for dual-stack > curl applications, and I am not sure that I fully understand your objections > like " It might in your situation, but it wouldn't in everybody else's.". My point is that adding a callback doesn't fix anything on its own. An application would need to add a function to be called back, and so far, your application is the only one that would currently use it. I suspect there is a better way to solve this issue. For example, you didn't answer this question from my last message: If Curl_ipv6works() were not called in the CURL_IPRESOLVE_V6 case, would that solve the issues that are remaining? Since you're calling it a regression, then where did that regression occur? Was it in libcurl or in the kernel? Maybe this problem needs to be solved elsewhere. Dan -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
> Is that what you're referring to in all cases? Yes, all my "regressions" and "problems" referrals are for 30+ ms connection delays caused by Curl_ipv6works() used in the dual-stack IPv6-enabled libcurl. And these problems and regressions occur when code is migrated from IPv4 libcurl to dual-stack IPv6-enabled one. I never mentioned any other problems or regressions in this e-mail thread. > It might in your situation, but it wouldn't in everybody else's. It will help in my situation, but it might also help in somebody's else case as well. There are few points here: 1. These regressions happen for some pretty recent linux kernel versions and configurations used in embedded software, and somebody else might step on the same issues as I did when migrating to dual-stack libcurl and try to find a solution for the same dual-stack issues/regressions. 2. The Curl_ipv6works() is a good default approach, but it is very difficult to find a "one size that fits all" solution. That's why I don't propose that libcurl should provide a solution for everybody which covers all possible scenarios and corner cases. Instead, I propose that libcurl would provide a way for everyone to solve these issues if the system calls inside Curl_ipv6works() are causing problems critical for them. 3. I don't see any other good ways to work around these issues for IPv6-related resolve modes (CURL_IPRESOLVE_WHATEVER and CURL_IPRESOLVE_IPV6.) except letting curl application to provides a custom way of detecting if "IPv6 works". I think that my proposal will provide useful customization for dual-stack curl applications, and I am not sure that I fully understand your objections like " It might in your situation, but it wouldn't in everybody else's.". There are always cases where something will not be good enough, and additional customization helps to handle such cases. So, again, I am not proposing some concrete change to the Curl_ipv6works() which will just help my individual problem. Instead, I am proposing a dual-stack related customization which will help to solve issues with the default "IPv6 works" detection that may be very different from my specific problems, but still critical for someone's else application. Thanks, Dmitry Karpov -Original Message- From: curl-library On Behalf Of Dan Fandrich via curl-library Sent: Wednesday, September 21, 2022 1:53 PM To: curl-library@lists.haxx.se Cc: Dan Fandrich Subject: Re: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system On Wed, Sep 21, 2022 at 08:19:46PM +, Dmitry Karpov via curl-library wrote: > Daniel's PR fixed the issue only for CURL_IPRESOLVE_V4, but not for > CURL_IPRESOLVE_WHATEVER and CURL_IPRESOLVE_IPV6. > Even with his PR, both CURL_IPRESOLVE_WHATEVER and CURL_IPRESOLVE_IPV6 will > use Curl_ipv6works() and thus face regressions. I find many of your messages too vague, and I can't tell any more what exactly are the specific problems that you're still having. They're using terms like "regressions" and "problem" but I can't tell any more if you're referring to the 30 msec delay during Curl_ipv6works() or if there are other issues you're seeing. Is that what you're referring to in all cases? > Before Daniel's PR, the problem was for all three resolve modes: > CURL_IPRESOLVE_WHATEVER, CURL_IPRESOLVE_IPV6 and CURL_IPRESOLVE_IPV6 > > After his PR, the problem remains for two modes: > CURL_IPRESOLVE_WHATEVER and CURL_IPRESOLVE_IPV6 If Curl_ipv6works() were not called in the CURL_IPRESOLVE_V6 case, would that solve the issues that are remaining? > My proposal, in addition to Daniel's PR allows to close the gap and avoid > connection regressions for all three modes. It might in your situation, but it wouldn't in everybody else's. Dan -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
On Wed, Sep 21, 2022 at 08:19:46PM +, Dmitry Karpov via curl-library wrote: > Daniel's PR fixed the issue only for CURL_IPRESOLVE_V4, but not for > CURL_IPRESOLVE_WHATEVER and CURL_IPRESOLVE_IPV6. > Even with his PR, both CURL_IPRESOLVE_WHATEVER and CURL_IPRESOLVE_IPV6 will > use Curl_ipv6works() and thus face regressions. I find many of your messages too vague, and I can't tell any more what exactly are the specific problems that you're still having. They're using terms like "regressions" and "problem" but I can't tell any more if you're referring to the 30 msec delay during Curl_ipv6works() or if there are other issues you're seeing. Is that what you're referring to in all cases? > Before Daniel's PR, the problem was for all three resolve modes: > CURL_IPRESOLVE_WHATEVER, CURL_IPRESOLVE_IPV6 and CURL_IPRESOLVE_IPV6 > > After his PR, the problem remains for two modes: > CURL_IPRESOLVE_WHATEVER and CURL_IPRESOLVE_IPV6 If Curl_ipv6works() were not called in the CURL_IPRESOLVE_V6 case, would that solve the issues that are remaining? > My proposal, in addition to Daniel's PR allows to close the gap and avoid > connection regressions for all three modes. It might in your situation, but it wouldn't in everybody else's. Dan -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
Daniel's PR fixed the issue only for CURL_IPRESOLVE_V4, but not for CURL_IPRESOLVE_WHATEVER and CURL_IPRESOLVE_IPV6. Even with his PR, both CURL_IPRESOLVE_WHATEVER and CURL_IPRESOLVE_IPV6 will use Curl_ipv6works() and thus face regressions. Code migrating from IPv4-single libcurl to IPv6 dual-stack, which uses the default CURL_IPRESOLVE_WHATEVER value will face the regression as well. My proposal allows to cover cases where the default Curl_ipv6works() behavior creates problems for CURL_IPRESOLVE_WHATEVER and CURL_IPRESOLVE_IPV6 and solve them by providing a better way to detect whether IPv6 works or not on the system. Before Daniel's PR, the problem was for all three resolve modes: CURL_IPRESOLVE_WHATEVER, CURL_IPRESOLVE_IPV6 and CURL_IPRESOLVE_IPV6 After his PR, the problem remains for two modes: CURL_IPRESOLVE_WHATEVER and CURL_IPRESOLVE_IPV6 My proposal, in addition to Daniel's PR allows to close the gap and avoid connection regressions for all three modes. Thanks, Dmitry Karpov -Original Message- From: curl-library On Behalf Of Dan Fandrich via curl-library Sent: Wednesday, September 21, 2022 1:05 PM To: curl-library@lists.haxx.se Cc: Dan Fandrich Subject: Re: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system On Wed, Sep 21, 2022 at 07:40:10PM +, Dmitry Karpov via curl-library wrote: > Yes, doing things "lazily" in Daniel's PR fixes the problem for explicit > CURL_IPRESOLVE_V4, which will not call Curl_ipv6works() anymore. > But the problem is still there for the CURL_IPRESOLVE_WHATEVER, which is the > default value for the CURLOPT_IPRESOLVE option. I'm clearly missing something. If Daniel's PR fixes the issue by setting CURL_IPRESOLVE_V4, then why not just set CURL_IPRESOLVE_V4 when needed? Why do you feel an additional callback mechanism is still needed? Dan -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
On Wed, Sep 21, 2022 at 07:40:10PM +, Dmitry Karpov via curl-library wrote: > Yes, doing things "lazily" in Daniel's PR fixes the problem for explicit > CURL_IPRESOLVE_V4, which will not call Curl_ipv6works() anymore. > But the problem is still there for the CURL_IPRESOLVE_WHATEVER, which is the > default value for the CURLOPT_IPRESOLVE option. I'm clearly missing something. If Daniel's PR fixes the issue by setting CURL_IPRESOLVE_V4, then why not just set CURL_IPRESOLVE_V4 when needed? Why do you feel an additional callback mechanism is still needed? Dan -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
Yes, doing things "lazily" in Daniel's PR fixes the problem for explicit CURL_IPRESOLVE_V4, which will not call Curl_ipv6works() anymore. But the problem is still there for the CURL_IPRESOLVE_WHATEVER, which is the default value for the CURLOPT_IPRESOLVE option. So, migration from IPv4-enabled single-stack libcurl to IPv6-enabled dual-stack libcurl will introduce this problem if the default CURLOPT_IPRESOLVE option is used. And for IPv4-enabled libcurl, there was no reason to set this option, so most of the migrated code (if not all) will use default CURL_IPRESOLVE_WHATEVER value and step on this issue. Also, when IPv6 or dual stack is really desired or needed, the Curl_ipv6works() may still be not the best choice, and a better approach for "IPv6 works" is needed to avoid artificial connection creation regressions caused by the default Curl_ipv6works(). For such cases, the "curl_global_init_ipv6() with callback" approach will be a good addition to Daniel's PR, which will help to close performance gaps between the single-stack and IPv6-enabled dual-stack libcurls. It will help applications to provide better and faster ways to detect whether IPv6 works or not on their systems and avoid unnecessary connection delays for CURL_IPRESOLVE_WHATEVER, which is also the default mode. Thanks, Dmitry Karpov -Original Message- From: curl-library On Behalf Of Daniel Stenberg via curl-library Sent: Tuesday, September 20, 2022 11:57 PM To: Daniel Stenberg via curl-library Cc: Daniel Stenberg Subject: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system On Wed, 21 Sep 2022, Daniel Stenberg via curl-library wrote: > Yes. It sounds like we should be able to fix this by "lazily" do the > check first when it is actually necessary to know. If set to > CURL_IPRESOLVE_V4 there's no need to know. Whatever we decide here, I think this is a good idea so I went ahead and made a pull-request with a change like this. See https://github.com/curl/curl/pull/9553 -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
On Wed, 21 Sep 2022, Daniel Stenberg via curl-library wrote: Yes. It sounds like we should be able to fix this by "lazily" do the check first when it is actually necessary to know. If set to CURL_IPRESOLVE_V4 there's no need to know. Whatever we decide here, I think this is a good idea so I went ahead and made a pull-request with a change like this. See https://github.com/curl/curl/pull/9553 -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
On Tue, 20 Sep 2022, Dan Fandrich via curl-library wrote: How doesn't setting CURLOPT_IPRESOLVE fix this? Is it because Curl_ipv6works() is being called somewhere anyway, and it adds a 30 msec delay? So, if libcurl eliminated that call in the CURL_IPRESOLVE_V4 case, would it fix your problem? Yes. It sounds like we should be able to fix this by "lazily" do the check first when it is actually necessary to know. If set to CURL_IPRESOLVE_V4 there's no need to know. -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
On Tue, Sep 20, 2022 at 11:49:08PM +, Dmitry Karpov via curl-library wrote: > No matter how difficult it is (and maybe not always possible) to do it in > multiple places, setting IP_RESOLVE option doesn't solve the problem which I > am trying to resolve via this proposal. How doesn't setting CURLOPT_IPRESOLVE fix this? Is it because Curl_ipv6works() is being called somewhere anyway, and it adds a 30 msec delay? So, if libcurl eliminated that call in the CURL_IPRESOLVE_V4 case, would it fix your problem? -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
No, it is not just an engineering issue, which can be solved by changing some easy handle settings in multiple places. No matter how difficult it is (and maybe not always possible) to do it in multiple places, setting IP_RESOLVE option doesn't solve the problem which I am trying to resolve via this proposal. The "IPv6 works" is checked when a multi handle is created, but before any easy handle is added to it, so it is not related to the IP_RESOLVE option set on any easy handle added to that multi handle later. This is a "system setting" which is checked and set only once when a multi handle is created. The "CURLOPT_USE_SSL, CURLOPT_FOLLOWLOCATION, CURLOPT_USERAGENT or tons of other options" are transfer-specific options managing what some specific transfer would like to use, whereas "IPv6 works" is a system setting, which essentially affects all handles used in the application. So, currently even if IP_RESOLVE is set to IPV4_ONLY for some easy handle, any transfer including transfer doing "easy transfer" via curl_easy_perform() (which creates a multi handle under the hood) will step on this regression when using IPv6-enabled dual stack libcurl vs IPv4 single stack libcurl. The default Curl_ipv6works() is a good default method to detect where IPv6 works or not on the system, but as any default - it is not the best method for ALL possible scenarios. That's why I see the need to provide additional flexibility allowing to detect whether IPv6 works or not in a better way than Curl_ipv6works() does. And again, this issue occurs regardless which IP_RESOLVE setting is used on some easy handle. Thanks, Dmitry Karpov -Original Message- From: Dan Fandrich Sent: Tuesday, September 20, 2022 2:57 PM To: Dmitry Karpov Subject: Re: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system On Tue, Sep 20, 2022 at 09:23:35PM +, Dmitry Karpov wrote: > That's exactly what I would like to avoid - going through a gazillion places > in numerous components and change something there just to improve how > Curl_ipv6works(). This argument could be made for a hundred other options. Other applications want to set CURLOPT_USE_SSL, CURLOPT_FOLLOWLOCATION, CURLOPT_USERAGENT or tons of other options on every handle, or CURLOPT_FTP_SKIP_PASV_IP, CURLOPT_TCP_KEEPALIVE or CURLOPT_DNS_LOCAL_IP6 based on local network conditions, but none of those make any sense to have set in a callback, either. It sounds to me like the underlying issue isn't really a technical one but rather a software engineering one, which patching libcurl isn't going to solve. Dan -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
> I still haven't seen any reason this needs to be a callback, besides that
> it's easier to integrate into your program.
> If you insist on this being a callback, then you can just call it yourself by
> changing your code from using
> curl_easy_init() everywhere to using dmitry_curl_easy_init():
That's exactly what I would like to avoid - going through a gazillion places in
numerous components and change something there just to improve how
Curl_ipv6works().
Unfortunately, my "application" is a quite large - with a lot of independent
components which are not easy or even feasible to change.
Some of the components using curl code are "3rd party" code integrated into my
application, which I just can't change, and which also have regressions caused
by default Curl_ipv6works() behavior.
So again, the "curl_global_init_ipv6() with callback" approach solves this
problem very easily and cleanly with minimal changes in the code and without
any need to touch "closed code" components, which just can't be modified.
Besides, Curl already has mechanism allowing applications to define global
memory management functions.
So, the "IPv6 works" callback will be just another application/system level
callback allowing to tune how the application should handle some "system" level
behavior.
Thanks,
Dmitry Karpov
-Original Message-
From: curl-library On Behalf Of Dan
Fandrich via curl-library
Sent: Tuesday, September 20, 2022 1:42 PM
To: libcurl development
Cc: Dan Fandrich
Subject: Re: [EXTERNAL] Re: Feature request: provide ability to set a global
callback function telling libcurl if IPv6 works on the system
On Tue, Sep 20, 2022 at 08:28:10PM +, Dmitry Karpov wrote:
> Yes, I want to use dual-stack in general. That's why my application has
> numerous components which use CURLOPT_IPRESOLVE = AUTO.
> But if IPv6 doesn't work on a "system level", I want my curl code to be as
> fast as "IPv4 only" resolve mode without changing anything in my code - like
> doing detection that IPv6 doesn't work and modifying IP resolve mode in all
> the places where I create and set an easy handle (that's your suggestion as I
> understood it).
I still haven't seen any reason this needs to be a callback, besides that it's
easier to integrate into your program. If you insist on this being a callback,
then you can just call it yourself by changing your code from using
curl_easy_init() everywhere to using dmitry_curl_easy_init():
CURL *dmitry_curl_easy_init(void) {
CURL *c=curl_easy_init();
my_ipv6_callback(c);
return c;
}
libcurl doesn't need a new callback mechanism to do this.
Dan
--
Unsubscribe: https://lists.haxx.se/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
--
Unsubscribe: https://lists.haxx.se/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
Re: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
On Tue, Sep 20, 2022 at 08:28:10PM +, Dmitry Karpov wrote:
> Yes, I want to use dual-stack in general. That's why my application has
> numerous components which use CURLOPT_IPRESOLVE = AUTO.
> But if IPv6 doesn't work on a "system level", I want my curl code to be as
> fast as "IPv4 only" resolve mode without changing anything in my code - like
> doing detection that IPv6 doesn't work and modifying IP resolve mode in all
> the places where I create and set an easy handle (that's your suggestion as I
> understood it).
I still haven't seen any reason this needs to be a callback, besides that it's
easier to integrate into your program. If you insist on this being a callback,
then you can just call it yourself by changing your code from using
curl_easy_init() everywhere to using dmitry_curl_easy_init():
CURL *dmitry_curl_easy_init(void) {
CURL *c=curl_easy_init();
my_ipv6_callback(c);
return c;
}
libcurl doesn't need a new callback mechanism to do this.
Dan
--
Unsubscribe: https://lists.haxx.se/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
> I don't see your argument. Either you want IPv6 for a connection or you don't. Yes, I want to use dual-stack in general. That's why my application has numerous components which use CURLOPT_IPRESOLVE = AUTO. But if IPv6 doesn't work on a "system level", I want my curl code to be as fast as "IPv4 only" resolve mode without changing anything in my code - like doing detection that IPv6 doesn't work and modifying IP resolve mode in all the places where I create and set an easy handle (that's your suggestion as I understood it). In other words, I want benefits of dual-stack when it works, and I don't want to pay any performance penalties when it doesn't. And I also don't want to pay code change penalties to modify CURLOPT_IPRESOLVE to IPv4 for cases when IPv6 doesn't work in ALL the places where easy handles are created and set up. > . A callback is only useful if the application needs to change something > while a transfer is ongoing The callback I am proposing is a global application callback, not a transfer-specific callback which, indeed, is needed only for transfer-in-progress operations. My callback will be set when application initializes curl via curl_global_init_ipv6() function, and it will be used when a multi handle is created (and before any easy handles are added to it) to set "IPv6 works" mode for the multi-handle . This is very similar to what Curl_ipv6works() currently does, only my proposal will allow to use application-specific function instead of Curl_ipv6works(). Thanks, Dmitry Karpov -Original Message- From: curl-library On Behalf Of Dan Fandrich via curl-library Sent: Tuesday, September 20, 2022 12:58 PM To: libcurl development Cc: Dan Fandrich Subject: Re: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system On Tue, Sep 20, 2022 at 07:24:42PM +, Dmitry Karpov wrote: > Not necessarily. > If we have a bunch of applications on the system with large codebases which > use CURLOPT_IPRESOLVE = AUTO in too many places, then all these places should > detect somehow that "IPv6 doesn't work" and change the resolve mode to IPv4. > It is just not always feasible or desirable. > > The "curl_global_init_ipv6() with callback" solves this problem very easily > and cleanly - as just one global callback can instantly tell all multi > handles created in the application that "IPv6 doesn't work" and let them all > fall back to IPv4. I don't see your argument. Either you want IPv6 for a connection or you don't. Just set the right value using curl_easy_getopt() before you call perform. A callback is only useful if the application needs to change something while a transfer is ongoing, which isn't the case for this option. -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
On Tue, Sep 20, 2022 at 07:24:42PM +, Dmitry Karpov wrote: Not necessarily. If we have a bunch of applications on the system with large codebases which use CURLOPT_IPRESOLVE = AUTO in too many places, then all these places should detect somehow that "IPv6 doesn't work" and change the resolve mode to IPv4. It is just not always feasible or desirable. The "curl_global_init_ipv6() with callback" solves this problem very easily and cleanly - as just one global callback can instantly tell all multi handles created in the application that "IPv6 doesn't work" and let them all fall back to IPv4. I don't see your argument. Either you want IPv6 for a connection or you don't. Just set the right value using curl_easy_getopt() before you call perform. A callback is only useful if the application needs to change something while a transfer is ongoing, which isn't the case for this option. -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
RE: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
> An option like CURLOPT_IPRESOLVE seems like a much cleaner solution. Not necessarily. If we have a bunch of applications on the system with large codebases which use CURLOPT_IPRESOLVE = AUTO in too many places, then all these places should detect somehow that "IPv6 doesn't work" and change the resolve mode to IPv4. It is just not always feasible or desirable. The "curl_global_init_ipv6() with callback" solves this problem very easily and cleanly - as just one global callback can instantly tell all multi handles created in the application that "IPv6 doesn't work" and let them all fall back to IPv4. Thanks, Dmitry -Original Message- From: curl-library On Behalf Of Dan Fandrich via curl-library Sent: Tuesday, September 20, 2022 12:14 PM To: curl-library@lists.haxx.se Cc: Dan Fandrich Subject: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system On Tue, Sep 20, 2022 at 06:58:31PM +, Dmitry Karpov via curl-library wrote: > To cover such a range of cases, curl application needs a more flexible > approach for detection whether IPv6 works on the "system level". > And I guess the "curl_global_init_ipv6() with callback" provides such > flexibility and allows to satisfy a wide range of application-specific needs. Without discussing of the merits of such an idea, a callback function doesn't seem like the right way to configure this in libcurl. An option like CURLOPT_IPRESOLVE seems like a much cleaner solution. Dan -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
On Tue, Sep 20, 2022 at 06:58:31PM +, Dmitry Karpov via curl-library wrote: > To cover such a range of cases, curl application needs a more flexible > approach for detection whether IPv6 works on the "system level". > And I guess the "curl_global_init_ipv6() with callback" provides such > flexibility and allows to satisfy a wide range of application-specific needs. Without discussing of the merits of such an idea, a callback function doesn't seem like the right way to configure this in libcurl. An option like CURLOPT_IPRESOLVE seems like a much cleaner solution. Dan -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
RE: Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
> Ah, ok then I was wrong. Yes, I use the same application with IPv6-enabled libcurl on different Linux kernel versions (both with IPv6 enabled and disabled on kernel level) for different embedded devices and some of them (pretty recent 4.x, actually) show IPv6 socket delay failures. Some kernel configurations show delays if IPv6 is not enabled in the kernel and some if IPv6 is not set on the network interface. These kernels come from SOC vendors, and I can't debug and modify them to make IPv6 socket calls fail more quickly. > Another approach could then perhaps be to make curl deal with such failures > in a better/fallback way to avoid needing the dynamic "ipv6-works" flag ? Not sure, if I understand how curl can deal better with such problems. The "IPv6 works" check (if it returns "false") allows to avoid Happy Eyeballs timeout delays during dual-stack connection establishment for cases when IPv6 doesn't work on a system level. It disables dual-stack on the multi handle and all connections are established using IPv4 afterwards. And the "system level" may cover a quite wide range of cases - IPv6 is disabled on kernel level, IPv6 address is disabled on network interface(s), IPv6 connectivity check failed and system disabled in some configuration parameters etc. To cover such a range of cases, curl application needs a more flexible approach for detection whether IPv6 works on the "system level". And I guess the "curl_global_init_ipv6() with callback" provides such flexibility and allows to satisfy a wide range of application-specific needs. Thanks, Dmitry Karpov -Original Message- From: curl-library On Behalf Of Daniel Stenberg via curl-library Sent: Tuesday, September 20, 2022 12:05 AM To: Daniel F via curl-library Cc: Daniel Stenberg Subject: [EXTERNAL] Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system On Tue, 20 Sep 2022, Daniel F via curl-library wrote: > This is not true. Application can be built using curl with IPv6 > enabled (e.g. using RedHat 8 base docker image), then executed on > another system/machine with IPv6 disabled. If I remember correctly, > attempt to create IPv6 socket failed with error. Ah, ok then I was wrong. Another approach could then perhaps be to make curl deal with such failures in a better/fallback way to avoid needing the dynamic "ipv6-works" flag ? -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
On Tue, 20 Sep 2022, Daniel F via curl-library wrote: This is not true. Application can be built using curl with IPv6 enabled (e.g. using RedHat 8 base docker image), then executed on another system/machine with IPv6 disabled. If I remember correctly, attempt to create IPv6 socket failed with error. Ah, ok then I was wrong. Another approach could then perhaps be to make curl deal with such failures in a better/fallback way to avoid needing the dynamic "ipv6-works" flag ? -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
W dniu 2022-09-20 08:22, Daniel Stenberg via curl-library napisał(a): On Mon, 19 Sep 2022, Dmitry Karpov via curl-library wrote: Working with IPv6-enabled dual-stack libcurl, I noticed that for some Linux kernel configurations, it takes ~15-30ms more for dual-stack libcurl to establish IPv4 connections than for IPv4 single-stack libcurl. After looking into this issue, it turned out that the regression was caused by Curl_ipv6works() function enabled in IPv6-on builds. The origin for that IPv6 check was that back in the 90s it happened that configure detected IPv6 support but then at runtime curl couldn't actually use it. I dare to suggest that we don't see that kind of system setups and behavior anymore. I think maybe the time has come to make instead rather assume that it works when built IPv6 enabled. At least I think it is worth exploring that route first before we consider adding a new API function for it. This is not true. Application can be built using curl with IPv6 enabled (e.g. using RedHat 8 base docker image), then executed on another system/machine with IPv6 disabled. If I remember correctly, attempt to create IPv6 socket failed with error. -- Regards, Daniel -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
On Mon, 19 Sep 2022, Dmitry Karpov via curl-library wrote: Working with IPv6-enabled dual-stack libcurl, I noticed that for some Linux kernel configurations, it takes ~15-30ms more for dual-stack libcurl to establish IPv4 connections than for IPv4 single-stack libcurl. After looking into this issue, it turned out that the regression was caused by Curl_ipv6works() function enabled in IPv6-on builds. The origin for that IPv6 check was that back in the 90s it happened that configure detected IPv6 support but then at runtime curl couldn't actually use it. I dare to suggest that we don't see that kind of system setups and behavior anymore. I think maybe the time has come to make instead rather assume that it works when built IPv6 enabled. At least I think it is worth exploring that route first before we consider adding a new API function for it. -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Feature request: provide ability to set a global callback function telling libcurl if IPv6 works on the system
Hi All,
Working with IPv6-enabled dual-stack libcurl, I noticed that for some Linux
kernel configurations, it takes ~15-30ms more for dual-stack libcurl to
establish IPv4 connections than for IPv4 single-stack libcurl.
After looking into this issue, it turned out that the regression was caused by
Curl_ipv6works() function enabled in IPv6-on builds.
This function is called on every multi handle initialization and it uses
successful IPv6 socket creation call as an indicator that IPv6 actually works
on the system.
And in some cases, it takes ~15-30ms for IPv6 socket creation to fail, thus
creating IPv4 connection delay.
I propose to define a global callback which libcurl client application can use
to tell libcurl whether IPv6 works on the system or not - thus allowing to
eliminate delays caused by failed IPv6 socket creations.
And client app can set this callback in a special flavor of
curl_global_init_xxx() function (i.e. curl_global_init_ipv6()) similar to
curl_global_init_mem().
So, the prototype may look like:
/*
* callback function for checking if IPv6 is enabled on the system or
* for the application.
*
*/
typedef enum {
CURLIPV6_NO = 0, /* ipv6 is disabled and shouldn't be used. */
CURLIPV6_YES,/* ipv6 is enabled and can be used. */
CURLIPV6_AUTO/* use default libcurl mechanism to detect if IPv6 works */
} CURLipv6;
typedef CURLipv6(*curl_ipv6works_callback)();
/*
* NAME curl_global_init_ipv6()
*
* DESCRIPTION
*
* curl_global_init() or curl_global_init_ipv6() should be invoked exactly once
* for each application that uses libcurl. This function can be used to
* initialize libcurl and set user defined callback checking if IPv6 works
* on the system.
* Users can implement IPv6 checking routines to tell libcurl if it can use
* IPv6 in its transfers.
*/
CURL_EXTERN CURLcode curl_global_init_ipv6(long flags,
curl_ipv6works_callback ipv6_works_cb);
This approach is flexible enough to allow clients to specify special conditions
where they know when they do or don't want to use IPv6 with conditions where
they want to rely on libcurl default mechanism.
Example:
CURLipv6 check_ipv6() {
if (client_disables_ipv6) // We know that IPv6 doesn't work here.
return CURLipv6::CURLIPV6_NO;
// Use default check when we are not sure if IPv6 works or not.
return CURLipv6::CURLIPV6_AUTO;
}
// Init libcurl with "IPv6 works" callback.
curl_global_init_ipv6(CURL_GLOBAL_ALL, check_ipv6);
I am also attaching a patch file implementing this approach in libcurl.
Thanks,
Dmitry Karpov
curl_global_init_ipv6-global-function.patch
Description: curl_global_init_ipv6-global-function.patch
--
Unsubscribe: https://lists.haxx.se/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
