[lftp] Redirect Bug

[Nate Sutton]

Hi there,

I think I've found a small bug in the latest lftp. When I run `pget -n 5 '
https://example.com/foo'` and it gets redirected with a Location header
'/bar/baz' then the URL becomes 'https://example.com//bar/baz' -- notice
the double slash? Seems like a bug?
___
lftp mailing list
lftp@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp

[lftp] HTTP Basic Auth

[Nate Sutton]

Something changed between lftp 4.4.8 and lftp 4.8.4 with HTTP basic auth.
In the newer version it only sends the Authorization header if it receives
a challenge response from making a request without the header. In the older
version it always sent the Authorization header.

Is there a way to force sending the Authorization header on the first
request? Not all endpoints that accept HTTP basic auth respond with a
challenge, some redirect to oauth or whatever else.

Thanks!
___
lftp mailing list
lftp@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp

[lftp] LFTP Server

[Nate Sutton]

Have you ever considered making LFTP a server process? As in, you could
send commands to a shared server process that executes the transfers and
streams progress back the the client? In theory multiple processes could
initiate transfers and re-use connections.

For that matter, have you considered making machine-parseable output an
option? Maybe JSON like:

{"type": "progress", "progress": 10, "bytes_transferred": 1,
"bytes_total": 10, rate: 1}
(where progress is in percent and rate is in K/s)

And then maybe something like this when it finishes?
{"type": "complete", "result": "error", "error": "transfer aborted"}
___
lftp mailing list
lftp@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp

[lftp] LFTP manpages on OSX are busted

[Nate Sutton]

On lftp 4.8.0 installed via homebrew on OSX 10.11.6, I get a bunch of
errors trying to load the manpage and I think it prevents some information
from being displayed. The errors look like this:

/usr/bin/tbl::171: unrecognised format `<'
/usr/bin/tbl::171: giving up on this table
/usr/bin/tbl::190: `p' modifier must be followed by number
/usr/bin/tbl::190: unrecognised format `h'
/usr/bin/tbl::190: giving up on this table
/usr/bin/tbl::220: unrecognised format `h'
/usr/bin/tbl::220: giving up on this table
/usr/bin/tbl::243: unrecognised format `\'
/usr/bin/tbl::243: giving up on this table
/usr/bin/tbl::303: unrecognised format `\'
/usr/bin/tbl::303: giving up on this table
/usr/bin/tbl::323: unrecognised format `\'
/usr/bin/tbl::323: giving up on this table
/usr/bin/tbl::364: unrecognised format `\'
/usr/bin/tbl::364: giving up on this table
/usr/bin/tbl::398: unrecognised format `\'
/usr/bin/tbl::398: giving up on this table
/usr/bin/tbl::427: unrecognised format `\'
/usr/bin/tbl::427: giving up on this table
/usr/bin/tbl::454: unrecognised format `\'
/usr/bin/tbl::454: giving up on this table
/usr/bin/tbl::484: unrecognised format `\'
/usr/bin/tbl::484: giving up on this table
/usr/bin/tbl::545: unrecognised format `\'
/usr/bin/tbl::545: giving up on this table
/usr/bin/tbl::571: unrecognised format `\'
/usr/bin/tbl::571: giving up on this table

Looking at the source of lftp.1, the errors don't seem to match up to the
text.

I don't understand the manpage format but I suspect OSX is using a crappy,
old tbl, like it does with other tools. `tbl -v` shows "GNU tbl (groff)
version 1.19.2". I'm not sure if this is something you're interesting in
fixing but it would be nice to have the full manpage on osx.
___
lftp mailing list
lftp@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp

[lftp] FTPS slowed down at 4.5

[Nate Sutton]

Hi there,

I recently upgraded lftp from 4.2.2 to 4.6.5 and I noticed that FTPS slowed
down quite a bit. I bisected versions and determined that the slowdown
began at 4.5.0 and continues through 4.7.1. With 4.4.16 I'm peaking at 53
M/s downloads (with pget -n 5) with this server but with 4.5.0 I'm peaking
at ~330 K/s (again, with pget -n 5) with the same file and server.

This isn't expected, right? I can't see anything in the debugging info
that's different between 4.7.1 and 4.2.2 to explain the difference in
speeds and I'm using the same options:

set dns:order inet
set xfer:clobber true
set ftp:use-quit false
set net:max-retries 3
set ssl:verify-certificate no
set ftp:ssl-allow true
set ftp:ssl-force true
set ftp:ssl-protect-list true
set ftp:ssl-protect-data true
set ftp:ssl-protect-fxp true
set ftps:initial-prot ""

Any idea on things I can try to improve the throughput?
___
lftp mailing list
lftp@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp

Re: [lftp] misbehaving HTTP server makes lftp retry forever

[Nate Sutton]

A that makes sense, and that'll work, thanks!

On Thu, Apr 14, 2016 at 5:36 AM, Alexander V. Lukyanov <l...@netis.ru> wrote:

> On Thu, Apr 14, 2016 at 01:50:35AM -0500, Nate Sutton wrote:
> > I tried it and setting it to a low timeout setting (10) made it fail
> with a
> > timeout, so it does work for the purpose, but if possible I'd like to
> just
> > limit the number of retries. Is there a setting I'm missing, maybe?
>
> It should not be set to a low value, the default is one day. The timer is
> reset every time when the transfer goes over "high watermark". So when the
> transfer starts over and does not reach the high watermark for the
> specified in xfer:timeout time, the transfer times out and fails.
>
> --
>Alexander.
>
> > On Wed, Apr 13, 2016 at 8:46 PM, Nate Sutton <nathan.sut...@gmail.com>
> > wrote:
> >
> > > What should I be setting this to? I don't quite understand how setting
> a
> > > timeout will fix this issue, could you explain? lftp isn't stalling for
> > > very long while downloading and intermittent stalls in transfers ought
> to
> > > be tolerated, so I wouldn't want to set it too low. And besides, the
> error
> > > message "recv: SSL_read: error::lib(0):func(0):reason(0)"
> implies
> > > that the peer shut down the connection, right? It seems in this case
> that
> > > limiting the number of retries when the peer terminates the connection
> > > would be better, since otherwise it gets that error message and
> retries the
> > > transfer again and again forever.
> > >
> > > Thoughts?
> > >
> > > On Wed, Apr 13, 2016 at 5:00 AM, Alexander V. Lukyanov <l...@netis.ru>
> > > wrote:
> > >
> > >> On Tue, Mar 29, 2016 at 07:37:06PM -0500, Nate Sutton wrote:
> > >> > I'm downloading from an HTTP server where it stops sending the file
> at
> > >> > about 40 MB and then stops sending data. lftp doesn't seem to handle
> > >> this
> > >> > very well and retries the transfer over and over and over,
> indefinitely.
> > >> >
> > >> > https://gist.github.com/nate/e7e47d3a86118439c36a1877847d8b01
> > >> >
> > >> > This happens with pget -n 5, pget -n 1, and get.
> > >> >
> > >> > Is this expected behavior? Is there a way to disable it? Or set a
> max
> > >> > number of retries? net:max-retries is already set to 3...
> > >>
> > >> Please test
> > >>
> > >>
> https://github.com/lavv17/lftp/commit/989b10e928d73136ef2460572350f14a6a6d14c7
> > >> or http://lftp.yar.ru/ftp/devel/lftp-4.7.1.8-a7ceb-dirty.tar.gz
> > >> which introduces new xfer:timeout setting.
> > >>
> > >> --
> > >>Alexander.
> > >>
> > >
> > >
>
> > ___
> > lftp mailing list
> > lftp@uniyar.ac.ru
> > http://univ.uniyar.ac.ru/mailman/listinfo/lftp
>
___
lftp mailing list
lftp@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp

Re: [lftp] misbehaving HTTP server makes lftp retry forever

[Nate Sutton]

What should I be setting this to? I don't quite understand how setting a
timeout will fix this issue, could you explain? lftp isn't stalling for
very long while downloading and intermittent stalls in transfers ought to
be tolerated, so I wouldn't want to set it too low. And besides, the error
message "recv: SSL_read: error::lib(0):func(0):reason(0)" implies
that the peer shut down the connection, right? It seems in this case that
limiting the number of retries when the peer terminates the connection
would be better, since otherwise it gets that error message and retries the
transfer again and again forever.

Thoughts?

On Wed, Apr 13, 2016 at 5:00 AM, Alexander V. Lukyanov <l...@netis.ru> wrote:

> On Tue, Mar 29, 2016 at 07:37:06PM -0500, Nate Sutton wrote:
> > I'm downloading from an HTTP server where it stops sending the file at
> > about 40 MB and then stops sending data. lftp doesn't seem to handle this
> > very well and retries the transfer over and over and over, indefinitely.
> >
> > https://gist.github.com/nate/e7e47d3a86118439c36a1877847d8b01
> >
> > This happens with pget -n 5, pget -n 1, and get.
> >
> > Is this expected behavior? Is there a way to disable it? Or set a max
> > number of retries? net:max-retries is already set to 3...
>
> Please test
>
> https://github.com/lavv17/lftp/commit/989b10e928d73136ef2460572350f14a6a6d14c7
> or http://lftp.yar.ru/ftp/devel/lftp-4.7.1.8-a7ceb-dirty.tar.gz
> which introduces new xfer:timeout setting.
>
> --
>Alexander.
>
___
lftp mailing list
lftp@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp

Re: [lftp] misbehaving HTTP server makes lftp retry forever

[Nate Sutton]

I tried it and setting it to a low timeout setting (10) made it fail with a
timeout, so it does work for the purpose, but if possible I'd like to just
limit the number of retries. Is there a setting I'm missing, maybe?

On Wed, Apr 13, 2016 at 8:46 PM, Nate Sutton <nathan.sut...@gmail.com>
wrote:

> What should I be setting this to? I don't quite understand how setting a
> timeout will fix this issue, could you explain? lftp isn't stalling for
> very long while downloading and intermittent stalls in transfers ought to
> be tolerated, so I wouldn't want to set it too low. And besides, the error
> message "recv: SSL_read: error::lib(0):func(0):reason(0)" implies
> that the peer shut down the connection, right? It seems in this case that
> limiting the number of retries when the peer terminates the connection
> would be better, since otherwise it gets that error message and retries the
> transfer again and again forever.
>
> Thoughts?
>
> On Wed, Apr 13, 2016 at 5:00 AM, Alexander V. Lukyanov <l...@netis.ru>
> wrote:
>
>> On Tue, Mar 29, 2016 at 07:37:06PM -0500, Nate Sutton wrote:
>> > I'm downloading from an HTTP server where it stops sending the file at
>> > about 40 MB and then stops sending data. lftp doesn't seem to handle
>> this
>> > very well and retries the transfer over and over and over, indefinitely.
>> >
>> > https://gist.github.com/nate/e7e47d3a86118439c36a1877847d8b01
>> >
>> > This happens with pget -n 5, pget -n 1, and get.
>> >
>> > Is this expected behavior? Is there a way to disable it? Or set a max
>> > number of retries? net:max-retries is already set to 3...
>>
>> Please test
>>
>> https://github.com/lavv17/lftp/commit/989b10e928d73136ef2460572350f14a6a6d14c7
>> or http://lftp.yar.ru/ftp/devel/lftp-4.7.1.8-a7ceb-dirty.tar.gz
>> which introduces new xfer:timeout setting.
>>
>> --
>>Alexander.
>>
>
>
___
lftp mailing list
lftp@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp

[lftp] HTTPS

[Nate Sutton]

Is there any way to get binaries from the lftp site over HTTPS?
___
lftp mailing list
lftp@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp

Re: [lftp] ftp: wait for QUIT reply before closing control socket

[Nate Sutton]

Ok, I'll just go with ftp:use-quit then. Thanks!

On Wed, Mar 30, 2016 at 2:33 AM, Alexander V. Lukyanov <l...@netis.ru> wrote:

> On Tue, Mar 29, 2016 at 06:25:27PM -0500, Nate Sutton wrote:
> > Is there a way to disable this? I just went from 4.2.2 to 4.6.5 and at
> > least one FTP server I integrate with doesn't always reply to QUIT
> > commands. I'm not sure of the implications of `set ftp:use-quit false` so
>
> I think `set ftp:use-quit false` is quite safe.
> You can also do `set net:timeout 1' before disconnecting.
>
> --
>Alexander.
>
___
lftp mailing list
lftp@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp

Re: [lftp] Setting up FTPS connection question...

[Nate Sutton]

How much data are you moving where the slight speed difference between SFTP
and FTPS will matter? How confident that you can correctly set up FTPS?
Setting up SFTP securely is easy and I value security over performance when
it comes to private files. If you want to improve SFTP perf you can do a
few things. First, if you're transferring files that are already compressed
(video, images, gzipped files) then you should try turning off Compression
in the config. Second, use pget with multiple connections and you might
find it's fast enough. Lastly, if those don't help enough, there's always
http://www.psc.edu/index.php/hpn-ssh


On Tuesday, December 1, 2015, Steven Dennis  wrote:

> Thanks so much. Got a little further but ran into this.
>
> pi@raspberrypi ~ $ lftp
> lftp :~> set ftp:ssl-force yes
> lftp :~> debug 5
> lftp :~> open ftp://@XX.XX.XX.XX -p21201
> Password:
>  Resolving host address...
>  1 address found: 37.48.65.81
> lftp @XX.XX.XX.XX:~> ls
>  Connecting to XX.XX.XX.XX (XX.XX.XX.XX) port 21201
> <--- 500 OOPS: prctl PR_SET_SECCOMP failed
> ---> FEAT
> Interrupt
>
> On Tue, Dec 1, 2015 at 3:00 AM, Alexander V. Lukyanov 
> wrote:
>
>> On Tue, Dec 01, 2015 at 02:46:16AM -0800, Steven Dennis wrote:
>> > lftp :~> open ftps://@XX.XX.XX.XX -p21201
>> >  gnutls_handshake: An unexpected TLS packet was received.
>> > ls: Fatal error: gnutls_handshake: An unexpected TLS packet was
>> received.
>>
>> ftps expects the connection to be encrypted from the very beginning. Most
>> probably you have set up ftp server so that the connection is plain text.
>> Use plain ftp:// and "set ftp:ssl-force yes" to make sure you are using
>> encrypted connection.
>>
>> --
>>Alexander.
>>
>
>
___
lftp mailing list
lftp@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp

[lftp] FTP upload bug?

[Nate Sutton]

When I try to upload to a host that has run out of storage capacity, it will 
keep retrying indefinitely. This is the debug output: 

-

 Closing data socket

copy: put rolled back to 0, seeking get accordingly 

--- PASV

--- 227 Entering Passive Mode (176,9,48,238,135,46). 

 Connecting data socket to (176.9.48.238) port 34606

 Data connection established 

--- STOR /12345.mp4

--- 150 Opening BINARY mode data connection for /12345.mp4 

--- 451 Transfer aborted. Input/output error 

 Closing data socket

copy: put rolled back to 0, seeking get accordingly

-

It just repeats forever and ever. Any idea how we might fix this? 

-- 
Nate Sutton


___
lftp mailing list
lftp@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp

Re: [lftp] multipart post

[Nate Sutton]

There are some RFCs and whatnot around this, if that's helpful:

http://www.w3.org/TR/html401/interact/forms.html#h-17.13.4.2
http://www.ietf.org/rfc/rfc1867.txt
http://www.ietf.org/rfc/rfc2388.txt

Specifying arbitrary headers would also help, too, if one wanted to upload to 
S3 using LFTP. 

On Thursday, August 23, 2012 at 1:31 PM, Nate Sutton wrote:

 An http post. Lftp will currently post the raw body to a URL that you give 
 it: 
 
 lftp -e 'set http:put-method POST;put /path/to/local/file -o 
 http://0.0.0.0:4567/upload;quit'
 
 I would like to perform a multipart post instead, where the file is treated 
 as though it is being posted with a form. 
 
 On Wednesday, August 22, 2012 at 9:22 PM, Alexander V. Lukyanov wrote:
 
  On Wed, Aug 22, 2012 at 04:30:50PM -0700, Nate Sutton wrote:
   Is there a way to multipart post with a file using lftp? 
  
  
  
  
  
  What do you mean by post?
  
  -- 
  Alexander.
  ___
  lftp mailing list
  lftp@uniyar.ac.ru (mailto:lftp@uniyar.ac.ru)
  http://univ.uniyar.ac.ru/mailman/listinfo/lftp
 



___
lftp mailing list
lftp@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp

Re: [lftp] multipart post

[Nate Sutton]

An http post. Lftp will currently post the raw body to a URL that you give it: 

lftp -e 'set http:put-method POST;put /path/to/local/file -o 
http://0.0.0.0:4567/upload;quit'

I would like to perform a multipart post instead, where the file is treated as 
though it is being posted with a form. 

On Wednesday, August 22, 2012 at 9:22 PM, Alexander V. Lukyanov wrote:

 On Wed, Aug 22, 2012 at 04:30:50PM -0700, Nate Sutton wrote:
  Is there a way to multipart post with a file using lftp? 
 
 
 
 What do you mean by post?
 
 -- 
 Alexander.
 ___
 lftp mailing list
 lftp@uniyar.ac.ru (mailto:lftp@uniyar.ac.ru)
 http://univ.uniyar.ac.ru/mailman/listinfo/lftp



___
lftp mailing list
lftp@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp

[lftp] multipart post

[Nate Sutton]

Is there a way to multipart post with a file using lftp? 


___
lftp mailing list
lftp@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp

[lftp] HTTP requests using the Range header

[Nate Sutton]

I can't find a way to perform an HTTP request in lftp that includes the Range 
header, is this possible? Is it possible to specify arbitrary headers? I 
imagine that would be useful as well.

Adding FTP range support would be nice as well, while I'm making feature 
requests. :)

-- 
Nate Sutton


___
lftp mailing list
lftp@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp

Re: [lftp] HTTP requests using the Range header

[Nate Sutton]

Awesome, thanks. Any thoughts on being able to use arbitrary HTTP headers? 

-- 
Nate Sutton


On Tuesday, November 8, 2011 at 11:31 AM, Alexander V. Lukyanov wrote:

 On Tue, Nov 08, 2011 at 11:17:20AM -0800, Nate Sutton wrote:
  I can't find a way to perform an HTTP request in lftp that includes the 
  Range header, is this possible? Is it possible to specify arbitrary 
  headers? I imagine that would be useful as well.
  
  Adding FTP range support would be nice as well, while I'm making feature 
  requests. :)
 
 See the command get1.
 
 -- 
 Alexander.



___
lftp mailing list
lftp@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp

Feature Request

[Nate Sutton]

I'm not sure if I've requested this before, but I'd love to see output from 
lftp that's easy to machine-parse. I wrap lftp in another application and 
parsing the output, while a fun challenge, could be made easier, particularly 
when there are errors or non-successful response codes.

Thanks!

--
Nate

SSL Verification on 4.2.0

[Nate Sutton]

In the latest LFTP (4.2.0) ssl:verify-certificate defaults to yes. This was 
surprising as I assumed (incorrectly) that it was on by default in previous 
versions. This doesn't seem to be an issue on the Linux systems we're using, 
and is a net positive IMO, but it fails on OSX 10.6.6. OSX stores the cert 
files in the Keychain application, apparently, so they can't be found on the 
file system. Does anyone know a way to get SSL verification working on OSX 
10.6.6 with LFTP 4.2.0?