Re: SF bug 1456 and associated commits

[Dan Fandrich]

On Thu, Dec 11, 2014 at 07:39:54PM +, Steve Holme wrote:
> My only objection is that we are then taking away the ability to allow the
> user to purposely send LF characters to the mail server for whatever reason
> - they may have a non RFC compliant mail server that requires the line
> ending to be LF or CR for example.

Plus, it takes away the ability to send a message directly that already has
CR/LF line endings, which I can imagine some SMTP workflows produce.
But, it does make some sense to have the default assume platform-native
end-of-line characters and perform protocol-required conversion as necessary.

>>> Dan
---
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Curl and GXS SFTP

[Marc Renault]

Hi,

I'm running into the same problem as a couple of posts:
http://curl.haxx.se/mail/lib-2014-09/0199.html
http://curl.haxx.se/mail/lib-2012-06/0156.html
http://curl.haxx.se/mail/lib-2012-03/0245.html
http://curl.haxx.se/mail/archive-2009-01/0114.html

Like the previous posts I'm trying to connect to GXS (OpenText) SFTP server
with Curl and I've run into the same issue.

Based on the previous posts, I did some digging to see what is going on
with the libssh2_sftp_stat_ex call at ssh.c:2089. From what I can tell, the
call works fine. But the flags that are set are only:
SSH_FILEXFER_ATTR_PERMISSIONS 0x0004
SSH_FILEXFER_ATTR_ACCESSTIME  0x0008
That is the value of attrs.flags is 12.

Out of curiosity, I hacked libSSH2 and forced it to read a value for the
file size. With a value greater than the size of the file, the file is
downloaded which makes sense.

Essentially, the code handles the case where there server doesn't respond
to the SSH_FXP_STAT but doesn't consider the case that a server might
respond to SSH_FXP_STAT without a file size. My proposed fix is the
following:

//ssh.c:2095
  else if(rc) {
/*
 * libssh2_sftp_open() didn't return an error, so maybe the server
 * just doesn't support stat()
 */
data->req.size = -1;
data->req.maxdownload = -1;
Curl_pgrsSetDownloadSize(data, -1);
  }
//PROPOSE FIX
  else if(attrs.filesize == 0)
{
  /*
   * The server doesn't return a file size with a stat().
   */
  data->req.size = -1;
  data->req.maxdownload = -1;
  Curl_pgrsSetDownloadSize(data, -1);
}
//END OF FIX
  else {

curl_off_t size = attrs.filesize;

I tested it against the GXS SFTP server (with the unhacked libSSH2) and it
works. I'm not nearly familiar enough with the rest of the code to have any
idea if this would mess anything else up.

Thanks,
Marc Renault
---
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

RE: SF bug 1456 and associated commits

[Steve Holme]

On Thu, 11 Dec 2014, Patrick Monnerat wrote:

> I'm about to commit a fix for SF bug 1456, but first, I request your
> opinion.

Please note there are two bugs listed in this bug report:

a) That curl doesn't handle LF to CRLF conversion
b) That dot stuffing doesn't work

The fixes I pushed for a) should fix any problems with b) and without that
fix there were certainly issues here.

I haven't been able to reproduce any other problems around b) now that the
fixes for a) are present, but the user has, and for that reason I re-opened
the bug report and await further feedback.

Any assistance with b) is certainly much appreciated from my point of view
;-)

> Forgive me Steve: you introduced CURLOPT_CRLF handling for the
> SMTP protocol, but I think the standards make this conversion
> mandatory (RFC 5321, 2.3.8).

No problem. I was just allowing the conversion to take place.

However, I think that the topic of "whether non CRLF line ending conversion
should be performed automatically by curl or not for certain protocols"
whilst related to the bug report is a different conversation and one that I
think should be explored here on the mailing list.

> Some objection ?
 
My only objection is that we are then taking away the ability to allow the
user to purposely send LF characters to the mail server for whatever reason
- they may have a non RFC compliant mail server that requires the line
ending to be LF or CR for example.

However, whichever way we decide to go I do think that if mandatory auto
conversion is to be performed then we should a) do the conversion for CR
line endings as well, b) review which other protocols we should do this for
and c) not perform the conversion in SMTP specific code - this could be
achieved by having a flag on the protocol handers that indicates the line
ending type for example.

Kind Regard

Steve

---
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Re: Multi interface and DNS load balancing

[Seppo Heikkilä]

>> DNS load balancing is quite common so this should work quite easily.
> It isn't easy. getaddrinfo() returns the addresses in a prio order and we
go
> through the addresses in that order. How are you suggesting we change this
> without overriding the users' /etc/gai.conf, RFC 3484 and whatever more
that
> need to be taken into account? Of course, involving c-ares complicates
matters
> slightly more too... I'm open for suggestions and patches on how to
improve this.

True, does not look that easy. Thanks for the explanation! My example had
also one bug with share handles. The code should have sharehandles[i] and
not sharehandles[0]. Thus assigning separate share handles seems to still
work also at least with c-ares.

Probably I keep my current solution, which is to resolve the hostname at
the beginning and assign IPs manually using CURLOPT_RESOLVE. So basically
having my own DNS cache layer in the end. An updated multi-app.c
demonstrating this approach is attached for those who might be interested.

Cheers,
Seppo

/***
 *  _   _   _
 *  Project ___| | | |  _ \| |
 * / __| | | | |_) | |
 *| (__| |_| |  _ <| |___
 * \___|\___/|_| \_\_|
 *
 * Copyright (C) 1998 - 2011, Daniel Stenberg, , et al.
 *
 * This software is licensed as described in the file COPYING, which
 * you should have received as part of this distribution. The terms
 * are also available at http://curl.haxx.se/docs/copyright.html.
 *
 * You may opt to use, copy, modify, merge, publish, distribute and/or sell
 * copies of the Software, and permit persons to whom the Software is
 * furnished to do so, under the terms of the COPYING file.
 *
 * This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
 * KIND, either express or implied.
 *
 ***/
/* This is an example application source code using the multi interface
 * and testing access to host that is using DNS load balancing.
 *
 * Compile:
 * g++ -lcurl -lcares multi-app.c
 *
 * Usage:
 * ./a.out | sort | uniq -c
 */

#include 
#include 

/* somewhat unix-specific */
#include 
#include 

/* curl stuff */
#include 
#include 
#include 

#define HANDLECOUNT 100   /* Number of simultaneous transfers */

int main(void) {
  CURL   *handles[HANDLECOUNT];
  CURLSH *sharehandles[HANDLECOUNT];
  CURLM  *multi_handle;

  int still_running; /* keep number of running handles */
  int i;

  curl_global_init(CURL_GLOBAL_ALL);

  // Print version info
  curl_version_info_data *d = curl_version_info(CURLVERSION_NOW);
  printf("Curl version: %s\n", d->version);
  if (d->features & CURL_VERSION_ASYNCHDNS) {
printf("ares enabled\n");
  } else {
printf("ares NOT enabled\n");
  }
  printf("C-ares: %s\n", ares_version(NULL));

  CURLMsg *msg;   /* for picking up messages with the transfer status */
  int msgs_left;  /* how many messages are left */

  /* init a multi stack */
  multi_handle = curl_multi_init();

  /* Allocate one CURL handle per transfer */
  for (i = 0; i < HANDLECOUNT; i++) {
handles[i] = curl_easy_init();
  }

//#define USE_SEPARATE_SHARE_HANDLES
#ifdef USE_SEPARATE_SHARE_HANDLES
  for (i = 0; i < HANDLECOUNT; i++) {
sharehandles[i] = curl_share_init();
curl_share_setopt(sharehandles[i], CURLSHOPT_SHARE, CURL_LOCK_DATA_DNS);
curl_easy_setopt(handles[i], CURLOPT_SHARE, sharehandles[i]);
  }
#endif

  // Force some example IPs (name resolve would be here)
  struct curl_slist *host1 = NULL;
  host1 = curl_slist_append(NULL, "google.com:80:173.194.40.74");
  CURLSH *host1sh = curl_share_init();
  curl_share_setopt(host1sh, CURLSHOPT_SHARE, CURL_LOCK_DATA_DNS);
  struct curl_slist *host2 = NULL;
  host2 = curl_slist_append(NULL,  "google.com:80:173.194.40.75");
  CURLSH *host2sh = curl_share_init();
  curl_share_setopt(host2sh, CURLSHOPT_SHARE, CURL_LOCK_DATA_DNS);

  for (i = 0; i < HANDLECOUNT; i++) {
//#define USE_PROXY
#ifdef USE_PROXY
curl_easy_setopt(handles[i], CURLOPT_PROXYPORT, 80L);
if (i % 2 == 1) {
  curl_easy_setopt(handles[i], CURLOPT_PROXY, "173.194.40.72");
} else {
  curl_easy_setopt(handles[i], CURLOPT_PROXY, "173.194.40.73");
}
#endif
#define USE_RESOLVE
#ifdef USE_RESOLVE
if (i % 2 == 1) {
  curl_easy_setopt(handles[i], CURLOPT_SHARE, host1sh);
  curl_easy_setopt(handles[i], CURLOPT_RESOLVE, host1);
} else {
  curl_easy_setopt(handles[i], CURLOPT_SHARE, host2sh);
  curl_easy_setopt(handles[i], CURLOPT_RESOLVE, host2);
}
#endif
curl_easy_setopt(handles[i], CURLOPT_UPLOAD, 0);
curl_easy_setopt(handles[i], CURLOPT_NOBODY, 1);
curl_easy_setopt(handles[i], CURLOPT_NOSIGNAL, 1L);
//curl_easy_setopt(handles[i], CURLOPT_DNS_CACHE_TIMEOUT, 0);
//curl_easy_setopt(handles[i],

SF bug 1456 and associated commits

[Patrick Monnerat]

 
I'm about to commit a fix for SF bug 1456, but first, I request your
opinion.
 
Forgive me Steve: you introduced CURLOPT_CRLF handling for the SMTP
protocol, but I think the standards make this conversion mandatory (RFC
5321, 2.3.8).
Thus the fix also reverts this handling and forces the LF conversion to
CRLF, regardless of the option setting.
To really stick to the standards, CR alone ought to be converted too
(but my fix doesn't).
 
Some objection ?
 
Patrick
---
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

long timeout after a lost connection (curl 7.39, windows)

[чи дер]

 Hi,

 I've come across the following curl problem, which can be easily reproduced on 
Windows (XP SP3 to be precise):

 1. connect to a remote host
 2. POST some data
 3. unplug the network cable (sometimes this happens when the server closes 
connection due to keep-alive settings, but unplugging is the most reliable)
 4. try POSTing another chunk of data and observe the following message:

"
* Found bundle for host XXX: 0x280e400
* Re-using existing connection! (#0) with host XXX
* Connected to XXX (XXX) port 8080 (#0)
> POST YYY HTTP/1.1
Host: XXX:8080
Accept: */*
Cookie: runmode=test
Content-Type: text/xml; charset="utf-8"
Content-Length: 940
* upload completely sent off: 940 out of 940 bytes
" This then stalls for minutes sometimes, while CURLOPT_TIMEOUT_MS and 
CURLOPT_CONNECTTIMEOUT_MS are set to 1 second. Such short timeout is critical 
in this specific application I'm working on.

If I break in the debugger it's sitting in:

Curl_poll(pollfd * ufds=0x02d1c6a0, unsigned int nfds=1, int timeout_ms=1000) 
Line 516 + 0x4f bytes C
curl_multi_wait(void * multi_handle=0x02851b18, curl_waitfd * 
extra_fds=0x, unsigned int extra_nfds=0, int timeout_ms=1000, int * 
ret=0x0305f500) Line 890 + 0x11 bytes C
easy_transfer(void * multi=0x02851b18) Line 684 + 0x16 bytes C
easy_perform(SessionHandle * data=0x02cd9018, bool events=bool_false) Line 800 
+ 0x18 bytes C
curl_easy_perform(void * easy=0x02cd9018) Line 819 + 0xb bytes C specifically 
in :

r = select((int)maxfd + 1,
#ifndef USE_WINSOCK
&fds_read, &fds_write, &fds_err,
#else
/* WinSock select() can't handle fd_sets with zero bits set, so
don't give it such arguments. See the comment about this in
Curl_check_socket().
*/
fds_read.fd_count ? &fds_read : NULL,
fds_write.fd_count ? &fds_write : NULL,
fds_err.fd_count ? &fds_err : NULL,
#endif
ptimeout);
Where only fds_read is set to 1 and has the original socket handle.
I understand this is a windows-specific problem and I was wondering if there's 
a way to resolve this maybe by changing some parameters of the socket when it 
is created?




---
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Re: List available curl options

[Dan Fandrich]

On Wed, Dec 10, 2014 at 10:52:46PM -0800, Jeroen Ooms wrote:
> I'm writing some bindings to libcurl for a scripting language. I was
> wondering if there is an API call or some other method to list the
> options (CURLOPT_* values and their integer) that the linked version
> of libcurl supports?
> 
> That would make it easier to test in the client if a particular option
> is available, and pass it to libcurl as the corresponding integer. I'm
> not sure how else to design a client where the user can specify an
> option (as a string), and then have it resolve to the appropriate
> CURLOPT value.
> 
> How are other clients doing this?

The curl_easy_setopt call will return CURLE_UNKNOWN_OPTION or
CURLE_NOT_BUILT_IN if the option is not supported. That's the most reliable way
of telling. It's unusual that a program would need to know if an option is
supported without actually trying to use that option.

>>> Dan
---
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html