Re: rsync over ssh fails with --files-from

2023-12-21 Thread Alex via rsync 
> The errors column is 0.  The drop column is 18.  The second bit number
> is the number of packets which should grow.  At least that is how I read
> it.  Column makes it more readable in a terminal but not so much in an
> email.
>

Yes, my apologies. I even debated inserting a screenshot. errs was
definitely the one I referenced, and it was increasing.

As it turns out, there was some kind of network configuration issue with my
network provider that caused this fscking problem.

I knew at some point it couldn't have been an rsync problem, but I also
wasn't sure if it was a local network problem (routing, problems with the
interface, number of open files, buffers, etc).

Now I can get on with more productive projects, ugh.

Thanks for everyone's help.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync over ssh fails with --files-from

2023-12-21 Thread Kevin Korb via rsync 
The errors column is 0.  The drop column is 18.  The second bit number 
is the number of packets which should grow.  At least that is how I read 
it.  Column makes it more readable in a terminal but not so much in an 
email.


On 12/21/23 14:18, Alex wrote:
Can someone help me determine if these errors are normal or if this 
could somehow be the cause? I've removed the last three columns for 
readability - they were all zeros.


# column -t /proc/net/dev
Inter-|      Receive    |        Transmit
face         |bytes     packets  errs      drop  fifo  frame  compressed 
  multicast|bytes  packets    errs    drop  fifo
lo:          133093161  146045   0         0     0     0      0 
   0                133093161  146045  0     0     0
ens18:       166999724  256655   0         16    0     0      0 
   0                167638513  218267  0     0     0


The "errs" for ens18 are steadily increasing, but the "drop" column is 
steady. It's also curious the errors on loopback would also be increasing?


Other ideas on how to troubleshoot this would be greatly appreciated.

Thanks,
Alex

On Wed, Dec 20, 2023 at 9:27 PM Alex > wrote:


Hi,

On Wed, Dec 20, 2023 at 11:03 AM Kevin Korb via rsync
mailto:rsync@lists.samba.org>> wrote:

What is the error?  I assume you know that with that syntax the
filelist.txt is local rather than remote.


Yes, I do know it refers to the list of local files.

There is no error - it just hangs indefinitely until some timeout
period. This is what it looks like on the remote side:

$ ps ax|grep rsync
  324075 ?        Ss     0:00 rsync --server -logDtpRe.LsfxCIvu
. /var/tmp/one/
  324094 ?        S      0:00 rsync --server -logDtpRe.LsfxCIvu
. /var/tmp/one/

On the local side, with a few - added to rsync, I see this:

recv_file_list done
get_local_name count=9 /var/tmp/one/
generator starting pid=324075
delta-transmission enabled
recv_generator(config1.cf ,0)
config1.cf  is uptodate
send_files(0, config1.cf )

then it just stalls until it eventually times out. However, if I
remove any one of the nine files from the filelist, it completes
normally.

It actually also exhibits the same problem without a filelist at all
- as long as I'm transferring multiple files, it fails. I suppose if
I were syncing a directory with less than nine files, it might
succeed, but local directory to any remote directory on this one
server fails.

This also isn't a disk space or inode problem or corrupt filesystem
problem - the same command works from a different host to this one
problematic host without a problem.

I've also confirmed the open file limit is large enough on both sides:

# ulimit -n
5

I've now spent hours and hours trying to isolate and troubleshoot
this problem. It really feels like it's somehow related to the
number of files being transferred, not the size of the files. If I
break up a directory into multiple attempts, I can eventually
transfer all the files (maybe 40 in total), but trying to send all
40 files at once and none get transferred.

It almost certainly has something to do with building the initial
list of files. If that list is too large, the transfer fails, even
if the majority of the files in the source are already in the
destination.

It also happens when I'm pushing the files to the destination or
pulling them from the remote to local.

Thanks,
Alex






On 12/20/23 09:50, Alex via rsync wrote:
 > Hi, I've been using rsync on fedora over ssh to sync
directories for
 > decades, but suddenly having a problem with transferring
multiple files
 > at a time to one specific host using --files-from. I can't
think of what
 > might have changed to have caused this. Using rsync to
transfer a single
 > file to this problematic host works successfully. It appears
to be
 > related to the number of files in the --files-from filelist.
More than
 > nine and it stalls; less than nine and it finishes
successfully. I'm
 > using the same version of rsync on both sides.
 >
 > (Server) Protocol versions: remote=31, negotiated=31
 > (Client) Protocol versions: remote=31, negotiated=31
 >
 > When running the following command, it appears to collect the
list of
 > files to be transferred, successfully makes the connection
with the
 > remote server, but then just stalls. Using strace on the
remote side
 > shows rsync appears to be waiting for data.
 >
 > rsync -a --files-from=/etc/mail/filelist.txt -e 'ssh -i
 > /root/keys/sync-key-v4'

Re: rsync over ssh fails with --files-from

2023-12-21 Thread Alex via rsync 
Can someone help me determine if these errors are normal or if this could
somehow be the cause? I've removed the last three columns for readability -
they were all zeros.

# column -t /proc/net/dev
Inter-|  Receive|Transmit
face |bytes packets  errs  drop  fifo  frame  compressed
 multicast|bytes  packetserrsdrop  fifo
lo:  133093161  146045   0 0 0 0  0   0
   133093161  146045  0 0 0
ens18:   166999724  256655   0 160 0  0   0
   167638513  218267  0 0 0

The "errs" for ens18 are steadily increasing, but the "drop" column is
steady. It's also curious the errors on loopback would also be increasing?

Other ideas on how to troubleshoot this would be greatly appreciated.

Thanks,
Alex

On Wed, Dec 20, 2023 at 9:27 PM Alex  wrote:

> Hi,
>
> On Wed, Dec 20, 2023 at 11:03 AM Kevin Korb via rsync <
> rsync@lists.samba.org> wrote:
>
>> What is the error?  I assume you know that with that syntax the
>> filelist.txt is local rather than remote.
>>
>
> Yes, I do know it refers to the list of local files.
>
> There is no error - it just hangs indefinitely until some timeout period.
> This is what it looks like on the remote side:
>
> $ ps ax|grep rsync
>  324075 ?Ss 0:00 rsync --server -logDtpRe.LsfxCIvu .
> /var/tmp/one/
>  324094 ?S  0:00 rsync --server -logDtpRe.LsfxCIvu .
> /var/tmp/one/
>
> On the local side, with a few - added to rsync, I see this:
>
> recv_file_list done
> get_local_name count=9 /var/tmp/one/
> generator starting pid=324075
> delta-transmission enabled
> recv_generator(config1.cf,0)
> config1.cf is uptodate
> send_files(0, config1.cf)
>
> then it just stalls until it eventually times out. However, if I remove
> any one of the nine files from the filelist, it completes normally.
>
> It actually also exhibits the same problem without a filelist at all - as
> long as I'm transferring multiple files, it fails. I suppose if I were
> syncing a directory with less than nine files, it might succeed, but local
> directory to any remote directory on this one server fails.
>
> This also isn't a disk space or inode problem or corrupt filesystem
> problem - the same command works from a different host to this one
> problematic host without a problem.
>
> I've also confirmed the open file limit is large enough on both sides:
>
> # ulimit -n
> 5
>
> I've now spent hours and hours trying to isolate and troubleshoot this
> problem. It really feels like it's somehow related to the number of files
> being transferred, not the size of the files. If I break up a directory
> into multiple attempts, I can eventually transfer all the files (maybe 40
> in total), but trying to send all 40 files at once and none get transferred.
>
> It almost certainly has something to do with building the initial list of
> files. If that list is too large, the transfer fails, even if the majority
> of the files in the source are already in the destination.
>
> It also happens when I'm pushing the files to the destination or pulling
> them from the remote to local.
>
> Thanks,
> Alex
>
>
>
>
>
>
>
>>
>> On 12/20/23 09:50, Alex via rsync wrote:
>> > Hi, I've been using rsync on fedora over ssh to sync directories for
>> > decades, but suddenly having a problem with transferring multiple files
>> > at a time to one specific host using --files-from. I can't think of
>> what
>> > might have changed to have caused this. Using rsync to transfer a
>> single
>> > file to this problematic host works successfully. It appears to be
>> > related to the number of files in the --files-from filelist. More than
>> > nine and it stalls; less than nine and it finishes successfully. I'm
>> > using the same version of rsync on both sides.
>> >
>> > (Server) Protocol versions: remote=31, negotiated=31
>> > (Client) Protocol versions: remote=31, negotiated=31
>> >
>> > When running the following command, it appears to collect the list of
>> > files to be transferred, successfully makes the connection with the
>> > remote server, but then just stalls. Using strace on the remote side
>> > shows rsync appears to be waiting for data.
>> >
>> > rsync -a --files-from=/etc/mail/filelist.txt -e 'ssh -i
>> > /root/keys/sync-key-v4' /etc/mail/tmp/ polaris:/var/tmp/one/
>> >
>> > sync-key-v4 is a passwordless ed25519 key, but I've tried a handful of
>> > other ed25519 keys.
>> >
>> > Could it be related to packet size or some kind of network disparity?
>> > It's not related to the size of the files, as I've tried large and
>> small
>> > and it doesn't matter - if the number of files exceeds 9, it fails.
>> >
>> >
>> >
>> >
>> >
>> >
>>
>> --
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>> Kevin Korb  Phone:(407) 252-6853
>> Systems Administrator   Internet:
>> FutureQuest, Inc.

Re: rsync over ssh fails with --files-from

2023-12-20 Thread Alex via rsync 
Hi,

On Wed, Dec 20, 2023 at 11:03 AM Kevin Korb via rsync 
wrote:

> What is the error?  I assume you know that with that syntax the
> filelist.txt is local rather than remote.
>

Yes, I do know it refers to the list of local files.

There is no error - it just hangs indefinitely until some timeout period.
This is what it looks like on the remote side:

$ ps ax|grep rsync
 324075 ?Ss 0:00 rsync --server -logDtpRe.LsfxCIvu .
/var/tmp/one/
 324094 ?S  0:00 rsync --server -logDtpRe.LsfxCIvu .
/var/tmp/one/

On the local side, with a few - added to rsync, I see this:

recv_file_list done
get_local_name count=9 /var/tmp/one/
generator starting pid=324075
delta-transmission enabled
recv_generator(config1.cf,0)
config1.cf is uptodate
send_files(0, config1.cf)

then it just stalls until it eventually times out. However, if I remove any
one of the nine files from the filelist, it completes normally.

It actually also exhibits the same problem without a filelist at all - as
long as I'm transferring multiple files, it fails. I suppose if I were
syncing a directory with less than nine files, it might succeed, but local
directory to any remote directory on this one server fails.

This also isn't a disk space or inode problem or corrupt filesystem problem
- the same command works from a different host to this one problematic host
without a problem.

I've also confirmed the open file limit is large enough on both sides:

# ulimit -n
5

I've now spent hours and hours trying to isolate and troubleshoot this
problem. It really feels like it's somehow related to the number of files
being transferred, not the size of the files. If I break up a directory
into multiple attempts, I can eventually transfer all the files (maybe 40
in total), but trying to send all 40 files at once and none get transferred.

It almost certainly has something to do with building the initial list of
files. If that list is too large, the transfer fails, even if the majority
of the files in the source are already in the destination.

It also happens when I'm pushing the files to the destination or pulling
them from the remote to local.

Thanks,
Alex







>
> On 12/20/23 09:50, Alex via rsync wrote:
> > Hi, I've been using rsync on fedora over ssh to sync directories for
> > decades, but suddenly having a problem with transferring multiple files
> > at a time to one specific host using --files-from. I can't think of what
> > might have changed to have caused this. Using rsync to transfer a single
> > file to this problematic host works successfully. It appears to be
> > related to the number of files in the --files-from filelist. More than
> > nine and it stalls; less than nine and it finishes successfully. I'm
> > using the same version of rsync on both sides.
> >
> > (Server) Protocol versions: remote=31, negotiated=31
> > (Client) Protocol versions: remote=31, negotiated=31
> >
> > When running the following command, it appears to collect the list of
> > files to be transferred, successfully makes the connection with the
> > remote server, but then just stalls. Using strace on the remote side
> > shows rsync appears to be waiting for data.
> >
> > rsync -a --files-from=/etc/mail/filelist.txt -e 'ssh -i
> > /root/keys/sync-key-v4' /etc/mail/tmp/ polaris:/var/tmp/one/
> >
> > sync-key-v4 is a passwordless ed25519 key, but I've tried a handful of
> > other ed25519 keys.
> >
> > Could it be related to packet size or some kind of network disparity?
> > It's not related to the size of the files, as I've tried large and small
> > and it doesn't matter - if the number of files exceeds 9, it fails.
> >
> >
> >
> >
> >
> >
>
> --
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> Kevin Korb  Phone:(407) 252-6853
> Systems Administrator   Internet:
> FutureQuest, Inc.   ke...@futurequest.net  (work)
> Orlando, Floridak...@sanitarium.net (personal)
> Web page:   https://sanitarium.net/
> PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync over ssh fails with --files-from

2023-12-20 Thread Kevin Korb via rsync 
What is the error?  I assume you know that with that syntax the 
filelist.txt is local rather than remote.


On 12/20/23 09:50, Alex via rsync wrote:
Hi, I've been using rsync on fedora over ssh to sync directories for 
decades, but suddenly having a problem with transferring multiple files 
at a time to one specific host using --files-from. I can't think of what 
might have changed to have caused this. Using rsync to transfer a single 
file to this problematic host works successfully. It appears to be 
related to the number of files in the --files-from filelist. More than 
nine and it stalls; less than nine and it finishes successfully. I'm 
using the same version of rsync on both sides.


(Server) Protocol versions: remote=31, negotiated=31
(Client) Protocol versions: remote=31, negotiated=31

When running the following command, it appears to collect the list of 
files to be transferred, successfully makes the connection with the 
remote server, but then just stalls. Using strace on the remote side 
shows rsync appears to be waiting for data.


rsync -a --files-from=/etc/mail/filelist.txt -e 'ssh -i 
/root/keys/sync-key-v4' /etc/mail/tmp/ polaris:/var/tmp/one/


sync-key-v4 is a passwordless ed25519 key, but I've tried a handful of 
other ed25519 keys.


Could it be related to packet size or some kind of network disparity? 
It's not related to the size of the files, as I've tried large and small 
and it doesn't matter - if the number of files exceeds 9, it fails.









--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html