[jira] [Commented] (SSHD-1173) SFTP worker threads got stuck while processing PUT methods against one specific SFTP server
[
https://issues.apache.org/jira/browse/SSHD-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17613818#comment-17613818
]
Kirk Rasmussen commented on SSHD-1173:
--
Adding this to my sshClient instance fixed it for me.
{code:java}
SftpModuleProperties.CHUNK_IF_WINDOW_LESS_THAN_PACKET.set(sshClient, true);
{code}
alternatively can use System property:
{code:java}
org.apache.sshd.config.sftp-chunk-if-window-less-than-packet=true {code}
This issue is referenced here as a server client compatibility problem but it
applies equally to client as well: [SSHD-1123] ChannelAsyncOutputStream breaks
downloads of sftp client by not chunking when the remote window is smaller than
the packet size - ASF JIRA (apache.org)
> SFTP worker threads got stuck while processing PUT methods against one
> specific SFTP server
> ---
>
> Key: SSHD-1173
> URL: https://issues.apache.org/jira/browse/SSHD-1173
> Project: MINA SSHD
> Issue Type: Question
>Affects Versions: 2.6.0
>Reporter: Pavel Pohner
>Priority: Critical
> Labels: SFTP, mina, sshd
>
> Hey guys,
> Recently I've encountered one remote SFTP server, that causes SFTP worker
> threads to enter TIMED_WAITING state, from which they do not recover. Please,
> take a look into this thread dump:
> {code:java}
> SFTP-worker-3 #2155 prio=5 os_prio=0 cpu=61692.09ms elapsed=1136151.86s
> tid=0x7fe41c005800 nid=0x18808 in Object.wait()
> [0x7fe145b1d000]SFTP-worker-3" #2155 prio=5 os_prio=0 cpu=61692.09ms
> elapsed=1136151.86s tid=0x7fe41c005800 nid=0x18808 in Object.wait()
> [0x7fe145b1d000] java.lang.Thread.State: TIMED_WAITING (on object
> monitor) at java.lang.Object.wait(java.base@11.0.10/Native Method) - waiting
> on at
> org.apache.sshd.common.future.DefaultSshFuture.await0(DefaultSshFuture.java:69)
> - waiting to re-lock in wait() <0x0006e3bbc420> (a
> org.apache.sshd.common.channel.IoWriteFutureImpl) at
> org.apache.sshd.common.future.AbstractSshFuture.verifyResult(AbstractSshFuture.java:109)
> at
> org.apache.sshd.common.io.AbstractIoWriteFuture.verify(AbstractIoWriteFuture.java:39)
> at
> org.apache.sshd.common.io.AbstractIoWriteFuture.verify(AbstractIoWriteFuture.java:30)
> at
> org.apache.sshd.common.future.VerifiableFuture.verify(VerifiableFuture.java:43)
> at
> org.apache.sshd.sftp.client.impl.DefaultSftpClient.send(DefaultSftpClient.java:292)
> at
> org.apache.sshd.sftp.client.impl.SftpOutputStreamAsync.flush(SftpOutputStreamAsync.java:210)
> at
> org.apache.sshd.sftp.client.impl.SftpOutputStreamAsync.write(SftpOutputStreamAsync.java:141)
> at java.io.InputStream.transferTo(java.base@11.0.10/InputStream.java:705) at
> com.mina.command.Put.replyInto(Put.java:55) at
> com.sftpserver.MinaSftpHandler.channelReadInternal(MinaSftpHandler.java:251)
> at
> com.sftpserver.MinaSftpHandler.lambda$channelRead$0(MinaSftpHandler.java:125)
> at com.sftpserver.MinaSftpHandler$$Lambda$392/0x000800764040.run(Unknown
> Source) at
> java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.10/Executors.java:515)
> at
> java.util.concurrent.FutureTask.run(java.base@11.0.10/FutureTask.java:264) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.10/ThreadPoolExecutor.java:1128)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.10/ThreadPoolExecutor.java:628)
> at java.lang.Thread.run(java.base@11.0.10/Thread.java:834)
> {code}
> Seems like the thread is waiting for lock in
> org.apache.sshd.common.future.DefaultSshFuture.await0(DefaultSshFuture.java),
> I'm currently not sure what thread holds the lock and why it's not ever
> released.
> Also, I'm not able to reproduce this, but it constantly happens to all the
> PUT methods targeting one specific remote server (which I don't own), so I
> suppose there would be something odd with the remote server, but it doesn't
> mean, that we shouldn't be able to deal with that.
> Have you ever seen such behavior? Could it be Mina sshd's bug? (seems like
> verify() in
> org.apache.sshd.sftp.client.impl.DefaultSftpClient.send(DefaultSftpClient.java:292)
> is called without any timeout, which then defaults to LONG.MAX here: at
> org.apache.sshd.common.future.VerifiableFuture.verify(VerifiableFuture.java:43),
> shouldn't we have configurable timeout here? or what's the point of default
> timeout? what are we really waiting for in at
> org.apache.sshd.common.future.DefaultSshFuture.await0(DefaultSshFuture.java:69)?)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org
For additional commands, e-mail: dev-h...@mina.apache.org
[jira] [Comment Edited] (SSHD-1173) SFTP worker threads got stuck while processing PUT methods against one specific SFTP server
[
https://issues.apache.org/jira/browse/SSHD-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17613590#comment-17613590
]
Kirk Rasmussen edited comment on SSHD-1173 at 10/6/22 3:08 PM:
---
I believe I may have also run into this running against MOVEIT transfer 2018
SP1 where transfers (11MB) are hanging in the same way as described above.
I've seen other areas where MOVEIT is buggy (e.g. canonicalPath("foo/bar.txt)
for a relative file) doesn't resolve correctly but directories do).
was (Author: JIRAUSER296597):
I believe I may have also run into this running against MOVEIT transfer 2018
SP1 where transfers (11MB) are hanging in the same way as described above.
> SFTP worker threads got stuck while processing PUT methods against one
> specific SFTP server
> ---
>
> Key: SSHD-1173
> URL: https://issues.apache.org/jira/browse/SSHD-1173
> Project: MINA SSHD
> Issue Type: Question
>Affects Versions: 2.6.0
>Reporter: Pavel Pohner
>Priority: Critical
> Labels: SFTP, mina, sshd
>
> Hey guys,
> Recently I've encountered one remote SFTP server, that causes SFTP worker
> threads to enter TIMED_WAITING state, from which they do not recover. Please,
> take a look into this thread dump:
> {code:java}
> SFTP-worker-3 #2155 prio=5 os_prio=0 cpu=61692.09ms elapsed=1136151.86s
> tid=0x7fe41c005800 nid=0x18808 in Object.wait()
> [0x7fe145b1d000]SFTP-worker-3" #2155 prio=5 os_prio=0 cpu=61692.09ms
> elapsed=1136151.86s tid=0x7fe41c005800 nid=0x18808 in Object.wait()
> [0x7fe145b1d000] java.lang.Thread.State: TIMED_WAITING (on object
> monitor) at java.lang.Object.wait(java.base@11.0.10/Native Method) - waiting
> on at
> org.apache.sshd.common.future.DefaultSshFuture.await0(DefaultSshFuture.java:69)
> - waiting to re-lock in wait() <0x0006e3bbc420> (a
> org.apache.sshd.common.channel.IoWriteFutureImpl) at
> org.apache.sshd.common.future.AbstractSshFuture.verifyResult(AbstractSshFuture.java:109)
> at
> org.apache.sshd.common.io.AbstractIoWriteFuture.verify(AbstractIoWriteFuture.java:39)
> at
> org.apache.sshd.common.io.AbstractIoWriteFuture.verify(AbstractIoWriteFuture.java:30)
> at
> org.apache.sshd.common.future.VerifiableFuture.verify(VerifiableFuture.java:43)
> at
> org.apache.sshd.sftp.client.impl.DefaultSftpClient.send(DefaultSftpClient.java:292)
> at
> org.apache.sshd.sftp.client.impl.SftpOutputStreamAsync.flush(SftpOutputStreamAsync.java:210)
> at
> org.apache.sshd.sftp.client.impl.SftpOutputStreamAsync.write(SftpOutputStreamAsync.java:141)
> at java.io.InputStream.transferTo(java.base@11.0.10/InputStream.java:705) at
> com.mina.command.Put.replyInto(Put.java:55) at
> com.sftpserver.MinaSftpHandler.channelReadInternal(MinaSftpHandler.java:251)
> at
> com.sftpserver.MinaSftpHandler.lambda$channelRead$0(MinaSftpHandler.java:125)
> at com.sftpserver.MinaSftpHandler$$Lambda$392/0x000800764040.run(Unknown
> Source) at
> java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.10/Executors.java:515)
> at
> java.util.concurrent.FutureTask.run(java.base@11.0.10/FutureTask.java:264) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.10/ThreadPoolExecutor.java:1128)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.10/ThreadPoolExecutor.java:628)
> at java.lang.Thread.run(java.base@11.0.10/Thread.java:834)
> {code}
> Seems like the thread is waiting for lock in
> org.apache.sshd.common.future.DefaultSshFuture.await0(DefaultSshFuture.java),
> I'm currently not sure what thread holds the lock and why it's not ever
> released.
> Also, I'm not able to reproduce this, but it constantly happens to all the
> PUT methods targeting one specific remote server (which I don't own), so I
> suppose there would be something odd with the remote server, but it doesn't
> mean, that we shouldn't be able to deal with that.
> Have you ever seen such behavior? Could it be Mina sshd's bug? (seems like
> verify() in
> org.apache.sshd.sftp.client.impl.DefaultSftpClient.send(DefaultSftpClient.java:292)
> is called without any timeout, which then defaults to LONG.MAX here: at
> org.apache.sshd.common.future.VerifiableFuture.verify(VerifiableFuture.java:43),
> shouldn't we have configurable timeout here? or what's the point of default
> timeout? what are we really waiting for in at
> org.apache.sshd.common.future.DefaultSshFuture.await0(DefaultSshFuture.java:69)?)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org
For additional commands, e-mail: dev-h...@mina.apache.org
[jira] [Commented] (SSHD-1173) SFTP worker threads got stuck while processing PUT methods against one specific SFTP server
[
https://issues.apache.org/jira/browse/SSHD-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17613590#comment-17613590
]
Kirk commented on SSHD-1173:
I believe I may have also run into this running against MOVEIT transfer 2018
SP1 where transfers (11MB) are hanging in the same way as described above.
> SFTP worker threads got stuck while processing PUT methods against one
> specific SFTP server
> ---
>
> Key: SSHD-1173
> URL: https://issues.apache.org/jira/browse/SSHD-1173
> Project: MINA SSHD
> Issue Type: Question
>Affects Versions: 2.6.0
>Reporter: Pavel Pohner
>Priority: Critical
> Labels: SFTP, mina, sshd
>
> Hey guys,
> Recently I've encountered one remote SFTP server, that causes SFTP worker
> threads to enter TIMED_WAITING state, from which they do not recover. Please,
> take a look into this thread dump:
> {code:java}
> SFTP-worker-3 #2155 prio=5 os_prio=0 cpu=61692.09ms elapsed=1136151.86s
> tid=0x7fe41c005800 nid=0x18808 in Object.wait()
> [0x7fe145b1d000]SFTP-worker-3" #2155 prio=5 os_prio=0 cpu=61692.09ms
> elapsed=1136151.86s tid=0x7fe41c005800 nid=0x18808 in Object.wait()
> [0x7fe145b1d000] java.lang.Thread.State: TIMED_WAITING (on object
> monitor) at java.lang.Object.wait(java.base@11.0.10/Native Method) - waiting
> on at
> org.apache.sshd.common.future.DefaultSshFuture.await0(DefaultSshFuture.java:69)
> - waiting to re-lock in wait() <0x0006e3bbc420> (a
> org.apache.sshd.common.channel.IoWriteFutureImpl) at
> org.apache.sshd.common.future.AbstractSshFuture.verifyResult(AbstractSshFuture.java:109)
> at
> org.apache.sshd.common.io.AbstractIoWriteFuture.verify(AbstractIoWriteFuture.java:39)
> at
> org.apache.sshd.common.io.AbstractIoWriteFuture.verify(AbstractIoWriteFuture.java:30)
> at
> org.apache.sshd.common.future.VerifiableFuture.verify(VerifiableFuture.java:43)
> at
> org.apache.sshd.sftp.client.impl.DefaultSftpClient.send(DefaultSftpClient.java:292)
> at
> org.apache.sshd.sftp.client.impl.SftpOutputStreamAsync.flush(SftpOutputStreamAsync.java:210)
> at
> org.apache.sshd.sftp.client.impl.SftpOutputStreamAsync.write(SftpOutputStreamAsync.java:141)
> at java.io.InputStream.transferTo(java.base@11.0.10/InputStream.java:705) at
> com.mina.command.Put.replyInto(Put.java:55) at
> com.sftpserver.MinaSftpHandler.channelReadInternal(MinaSftpHandler.java:251)
> at
> com.sftpserver.MinaSftpHandler.lambda$channelRead$0(MinaSftpHandler.java:125)
> at com.sftpserver.MinaSftpHandler$$Lambda$392/0x000800764040.run(Unknown
> Source) at
> java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.10/Executors.java:515)
> at
> java.util.concurrent.FutureTask.run(java.base@11.0.10/FutureTask.java:264) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.10/ThreadPoolExecutor.java:1128)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.10/ThreadPoolExecutor.java:628)
> at java.lang.Thread.run(java.base@11.0.10/Thread.java:834)
> {code}
> Seems like the thread is waiting for lock in
> org.apache.sshd.common.future.DefaultSshFuture.await0(DefaultSshFuture.java),
> I'm currently not sure what thread holds the lock and why it's not ever
> released.
> Also, I'm not able to reproduce this, but it constantly happens to all the
> PUT methods targeting one specific remote server (which I don't own), so I
> suppose there would be something odd with the remote server, but it doesn't
> mean, that we shouldn't be able to deal with that.
> Have you ever seen such behavior? Could it be Mina sshd's bug? (seems like
> verify() in
> org.apache.sshd.sftp.client.impl.DefaultSftpClient.send(DefaultSftpClient.java:292)
> is called without any timeout, which then defaults to LONG.MAX here: at
> org.apache.sshd.common.future.VerifiableFuture.verify(VerifiableFuture.java:43),
> shouldn't we have configurable timeout here? or what's the point of default
> timeout? what are we really waiting for in at
> org.apache.sshd.common.future.DefaultSshFuture.await0(DefaultSshFuture.java:69)?)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org
For additional commands, e-mail: dev-h...@mina.apache.org
